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INTRODUCTION 



Phase II of the Orthopaedic Training Study represents 
a landmark in the fullest sense of the word. It is the 
hope of the Study Staff that the information and experience 
gained in this, the first intensive study of medical speci- 
alty training, will serve as a model for others. However, 
the landmark indicates the beginning rather than the end of 
the trail. Even the beginning could not have been achieved 
without the cooperation of many representatives of the or- 
thopaedic community and the efforts of the research staff. 
Most notable among the former were the members of the Advi- 
sory Committee, Drs. George T. Aitken, Paul Curtiss, Charles 
Herndon, VJaltcr Hoyt, Jr., Paul Lipscomb and Fred C. Rey- 
nolds. It is to these men the Staff looked for guidance, 
they were never found wanting. 

The Final Report is divided into two volumes, the 
first briefly contains the following information: 

I. introduction and specific aims 

A. Background 

B. Methodology 

II. Nature and Variation of -- 



A. Programs 

B. Residents and attendings 

III. Analytic Study 



IV. Prediction Study 



The second volume, or Supplement, contains copies of 
most of the Study instruments, working papers, and report 
documents which will be helpful to those wishing to utilize 
the methodology developed. 
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The report: contains only n small portion of the mate- 
rials developed for the Study and the data generated through 
data collection efforts. Complete information is available 
from the Center for Educational Development provided the 
request is approved by the Advisory Committee. 
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INTRODUCTION AND SPECIFIC AIMS 



FOREWORD 



There has been a steady trend toward increased speciali- 
zation in medicine during the last 20 years. Parallel with, 
and indeed a part of this development has been increased em- 
phasis on formal board certification, both in the traditional 
specialities and the growing number of sub-specialities. Evi- 
dence of this can be seen in two areas: First, a general 

agreement that both servicer, at the community level have im- 
proved, both in quality and variety. Second, a growing con- 
cern that the potential practitioner is being required to 
invest greater periods of time in training, thus reducing his 
availability to deliver these health services. Concurrent 
with this concern is a question about the rigidity of time 
requirements which appear to deny an opportunity for individ- 
ual differences and the structuring of time requirements on 
some basis other than empiric. 

A general review of Board speciality requirements suggests 
that, while attempts are being made to apply our general know- 
ledge to differences in learning speed and patterns to training 
programs, a systematic effort to apply this knowledge had not 
been attempted. Finally, an empiric justification for the re- » 
structuring of both training programs and board requirements 
to allow for these differences could not be found in the 
research literature extant. 

It was discussions of this sort that led to an agreement 
between the American Board of Orthopaedic Surgery (ABOS) and 
the Center for Educational Development which culminated in a 
research project, "Efficient Use of Medical Manpower," funded 
by the Bureau of State Services. This Study was to focus on 
the definition and measurement of professional competence in 
orthopaedics, with the aim of increasing the validity and the 
reliability of such appraisals so that the Board could accept 
measures of competence as the primary criterion for certifica- 
tion. After reviewing the research completed and the resultant 
evaluation procedures developed from that research, the Board, 
in a meeting on June 30, 1967, agreed that it was time to begin 
exper imentnl modification of sequence, time, and content re- 
quirements on a controlled Study sample. 
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At. this same time the Skeletal Systems Committee of 
the National Research Council - National Academy of Science 
became concerned over the failure of orthopaedic training 
to produce academically oriented orthopno, 4 sts. It was 
apparent that these programs produced highly skilled ortho- 
paedists, many of whom became outstanding program directors 
but who had little familiarity with the vast body of know- 
ledge in education and with application to instructional 
practices in orthopaedics. 

On April 21, 1967, the NRC-NAS Skeletal Systems Com- 
mittee agreed to lend support to the* orthopaedic community 
for the* development and implementation of a study designed 
to examine time, sequence and content requirements of ex- 
isting orthopaedic programs. Specifically, the proposal, 
an extontion of Phase* I*, was designed to achieve the follow- 
ing broad objectives: 

1. To provide a model of individualized graduate 
education in medicine in which the demonstration 
of individual competence, rather than the fulfill- 
ment of rigid time and content requirements, mark 
the end point of formal training. 

2. To document the nature and variation of ortho- 
paedic training in the United States. 

3. To devise and test methods for increasing the 
efficiency and effectiveness of orthopaedic 
training. 

4. To determine the relationships between input 
training and output variables. 

5. To develop mechanisms that will facilitate con- 
tinuing institutional self-study of training 
programs. 



♦This report of Phase I is available from the Center 
for Educational Development. 
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6. To develop a pool of educational specialists 
in orthopaedics who can provide continuing 
leadership in the field. 



S tudy organizati on 



In order to achieve these objectives, a rather unique 
consortium was established with throe principal elements: 
a professional education staff, an advisory committee and 
a cadre of trained orthopaedic educators. Each of these 
groups provided a unique contribution to the Study. First, 
the professional education staff was able to supply an 
extensive knowledge of the science of education and research 
methodology to clarify questions and propose mechanisms for 
obtaining their answers. In addition, they provided the 
operating personnel for the Study, implementing plans, and 
providing coordination among the other groups involved in 
the Study. This strfff varied from one to four education 
professionals, the number dependant upon the particular 
portion of the Study being undertaken at the time. Second, 
an Advisory Committee* performed several unique services in 
addition to serving as representatives of the orthopaedic 
community and voicing their concerns. They provided broad 
range policies under which the Study operated, served as mon- 
itors of progress and direction, and finally, provided liaison 
with members of the orthopaedic community. The third group 
consisted of consultants. Under provisions of the Grant 
selected residents could substitute a portion of their resi- 
dency requirement by serving as consultants to the Ortho- 
paedic Training Study. This arrangement permitted them to 
gain expertise in the field of education while providing 
valuable subject matter input to the educational staff of 
the Study. Two men availed themselves of one-year fellowships 
and earned Masters' degrees in Medical Education. Two spent 
six months at the Center for Educational Development, while 
one spent approximately four months at the Center, and two 
others 30 days each. 



*The Advisory Committee consisted of two representa- 
tives from the Board, the Academy, and the MSC— NCR who were 
respectively Drs. Walter A. Hoyt, Jr. and Paul R. Lipscomb; 
Drs. George T. Aitken and Charles H. Herndon; and Drs. Fred 
Reynolds and Paul H. Curtiss, Jr. 
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Methods of Procedure 

The accomplishment of these goals within the proposed 
four-year study period required that several' phases of the 
Study be conducted simultaneously, but for clarity of pre- 
sentation, the elements of the Study will be described in- 
dependently. 

1. Introduction of Flexib i lity . Early data from the 
Orthopaedic In-Training Examination (OITE) had shown the 
wide range of achievement among residents in each year of 
training programs and the substantial overlap of achievement 
among individuals in all four years. The evidence suggested 
that some residents might qualify for certification in a 
significantly shorter time than was presently required even 
without modification of instructional content or methodology. 
In order to provide an early opportunity to demonstrate the 
validity of this hypothesis, 16 training programs, consisting 
of 280 residents* were designated as an intensive study 
group. The criteria for selection was developed jointly by 
an Advisory committee representing the three participating 
agencies. 

The cooperation of these program directors was enlisted 
to permit their residents, with the authorization of the 
America! Board of Orthopaedic Surgery, to depart from the 
present time and content distribution requirements without 
jeopardizing their eligibility for Board certification.* 

The participating residents were authorized to present an 
application for examination to the Board Committee on eligi- 
bility when, in the opinion of their program director, they 
were prepared to take the Board examination. Candidates who 
successfully completed the Board examination could proceed 
immediately to fulfill the practice requirement (presently 
}ne year) and, at the conclusion of that period, apply for 
final certification. 



* See Supplement. 
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In order to maximize the opportunity for increasing 
the speed and efficiency of learning, program directors 
■in the intensive study group were provided with assist- 
ance in introducing a process of continuous performance 
monitoring and giving feedback to the resident in order 
that both staff and resident might be aware of the individ- 
ual progress toward achievement of the critical components 
of competence in orthopaedics. 

The instruments to be used in this assessment included 
the annual In-Training Examinationgiven by the American 
Academy of Orthopaedic Surgery (developed by the Center, 
with the assistance of professional staff members from the 
B:>ard), and a variety of check lists, rating scales, and 
tests of complex cognitive and interpretive skills devel- 
oped in the original study. 

A second consideration in flexibility was the pro- 
posal for program modification that would .have the effect 
of introducint greater variety within a program in order 
to respond to differences in resident needs and interests. 

Residents in some programs were encouraged to sub- 
stitute a 6 to 12 month fellowship in research in medical 
education for more conventional rotations. Residents who 
took their Board exam earlier than us^al opted from among 
several alternatives.* 

2 . Intensive Study of Training Experiences . The 
critical components of competence in orthopaedic surgery 
defined the educational goals toward which training pro- 
grams should be directed.** The purpose of this phase 
of the Study identified the extent to which 1) training 
programs provide opportunities for residents to gain the 
defined knowledge, skills and attitudes, 2) the curricular 
organization, instructional materials and methods conform 
to generally accepted principles of learning and 3) system- 



*For a study of these early board candidates, see part 2, 
section 2. 

**Critical Components of Competency are discussed in the 
Final Report of Phase I. 
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atic evaluation is conducted and utilized for continuous 
program assessment. 

A stratified study sample (the intensive study group 
and the survey group*) of the 187 approved residency 
programs was selected to represent variations in academic 
affiliation, hopsital size, nature of population served, 
and geographic location. Each program was analyzed to 
document the following. 

A. Program organization - including schedule of resi- 
dent rotation, the personnel who supervise training, the 
facilities and resources to support the training. 

B. Program objectives - the mechanism created for 
establishment, review, and communication of the objectives 
to staff and residents. 

C. Program operation - activities and responsibil- 
ities of a resident sample, the nature or instructional 
procedures, of both a formal and informal nature, the 
nature of feedback to residents of their individual 
strengths and weaknesses as training progresses. 

D. Program evaluation - the mechanisms employed to 
accumulate data about resident progress,, program effec- 
tiveness and the utilization of these data in continuing 
program review. 

E. Program perceptions - identification of simi- 
larities among residents and staff in the perception of 
purposes, procedures and effectiveness. 

The variables (See Table IV - 1, p. 100) identified for 
objective 2 required diverse data, some of which already 
existed. Data included information on a resident's sex, 
marital status, academic background, internship, knowledge 
of science, ability at patient interviewing, etc. 



*The survey sample consisted of 35 resident training 
programs which received only the survey instruments. 
See supplement for a list of their programs. 
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Generally, the material which had to be gathered included 
information concerning background characteristics and 
training experiences with specific orthopaedic procedures. 
Type, location, and size of the program setting were areas 
of available institutional information while checklists, 
interviews, and questionnaires generated details of the 
program's mission or role in the community, association 
with other programs, room arrangement, support services 
(brace shops, physical therapy, special laboratories), etc. 

In addition to the general and specific characteristics of 
the program setting, other training variables for which 
information had to be gathered were the nature of resident 
responsibility for patient care, patterns of rotation, and 
professional qualifications of attendings. Knowledge, 
interpretive skills, and problem solving skills of the 
residents constituted the output variables and were measured 
by the Academy's In-Training Examination. 

The instruments* constructed to gather the above infor- 
mation were the following: 

The Program Description Questionnaire was designed to 
elicit information for classifying currently approved pro- 
grams according to affiliation, sources of financial support, 
administrative organization, primary commitment, orientation, 
physical facilities, clinical material, staff resources, 
size, program objectives and organization, education activ- 
ities, resident responsibilities and the like. This document 
was completed by 80 percent of the directors of all approved 
residency programs in orthopaedics in the United States.** 

The Institutional Description Form was designed to 
elicit detailed information from all institutions affili- 
ated with each program regarding the type of institution, 
primary mission, sources of financial support, administra- 
tive structure, educational resources, characteristics of 
the patient population served, interrelations among sub- 
specialty services, patient care facilities and supporting 





See Supplement for samples. 

e . 

** There was a 70 percent return on a re-distxibution " 
of this questionnaire in 1971. 
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services and decision-making process in screening patients 
for admission and directing their total health care. The 
form was completed by 78 percent of the directors of each 
institution affiliated with each program of the Study 
samples. 

The Resident Evaluation Form and The Candidate Eval- 
uation Form were designed to obtain preceptors ratings of 
residents and of candidates for certification, with respect 
to each of a number of performance factors related to the 
several components of competence identified as critical 
performance requirements. The Candidate Evaluation Form 
was completed by at least two persons who had known each 
resident over an extended period of time. The Resident 
Evaluation Form was completed by the chief. 

The Resident Evaluation of Operative Procedures was 
designed to identify the nature of the resident's experi- 
ence and the character of his responsibility with regard 
to various decision points in the operative management of 
patients and was returned by 90 percent of the residents 
in the Study. 

The Resident Procedures Form was designed to determine 
the resident's experience with technical procedures and his 
degree of confidence in ability to perform them. The form 
enabled identification of the setting in which he learned 
to perform a procedure and the nature of the supervision 
under which he learned it. The form listed 58 sample 
treatment and orthopaedic procedures and was completed by 
78 percent of residents in the Study sample. 

The Resident Time Log was designed to discover vari- 
ations in the work pattern in different orthopaedic residency 
programs. It was completed by 66 percent of residents in 
the Study samples at specified calendar periods selected 
to sample resident activities in each program. 

The Resident Attitude Survey was designed to identify 
both within and between program variations in the attitudes 
of residents toward patients, colleagues, instructional 
staff, physical and clinical facilities, and the profession 
and the training program. It was completed by 88 percent of 
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residents in the St\idy groups. 

The Resident Background Survey was designed to identify 
differences among residents in terms of their personal 1 ack- 
ground, education experience and attitudes. It was com- 
pleted by 98 percent of residents in the Study sample. 

The Attendin g Attitude Survey was designed to elicit 
attending staff attitudes toward program organization, 
teaching styles, resident behavior, interaction with resi- 
dents and with staff, etc. The survey was completed by 35 
percent of attendings in the Study. 

In addition to specifically designed instruments, 
data collection for this study employed other survey and 
observational techniques. In the first round of site 
visits, survey data was verified in an interview* with 
program chiefs and residents in training by requesting de- 
scriptive information on the educational resources actually 
\jsed in training, the nature and amount of instructional 
activity in which residents actually participate; the extent 
and character of resident responsibility for patient care, 
teaching, and research; the amount and character of feedback 
on performance regularly provided to each resident from 
senior residents, attending staff, and training chief. 
Observational data was obtained from site visits by the pro- 
ject staff. In the course of these two-day visits, a ran- 
dom sample of instructional experiences (rounds, conferences, 
seminars, operative and emergency room teaching) were de- 
scribed through carefully structured checklists and rating 
scales designed to document the quality of instructional 
exchange between trainees and their mentors, the educational 
goals (cognitive, psychomotor, effective) that the exchange 
is most likely to serve, and the pedagogic quality of the 
encounter (in terms of facilitation of learning, not the 
biomedical content) . 

3. Increasing Program Efficiency and Effectiveness . 
Through the intensive review described above, identification 
of areas in which new organization of training systems or 



* 

See Supplement for site visit procedures. 



utilization of alternative instructional modes would 
increase either a program's effectiveness of efficiency 
was made. Such changes included the following: 

Staff Development Programs . A brief seminar on 
graduate education designed for attending staff and some 
of the intensive study programs was conducted during the 
second round of site visits. Almost uniformly the attend- 
ings from these programs voiced a desire for additional 
materials in the general field of educational science. 

Many attendings commented on the fact that these sessions 
were the first time in which they had dealt with educational 
principles without regard for the particular subject matter 
and clinical considerations in the hospital setting. Ini- 
tially, two films illustrating the application (or violation) 
of fundamental principles of learning in characteristic edu- 
cational settings (lectures, conferences, rounds, operating 
room, and individual instruction) of residency training were 
prepared.* Their effectiveness resulted in the preparation 
of six additional films which were used on the subsequent 
site visits. These materials are available in the supplement. 

Instructional Innovation** . One of the components of 
program efficiency is the appropriate utilization of a 
teaching staff. This utilization should appropriate highly 
skilled staff members to the task for which they are trained 
and avoid use of staff to accomplishing routine tasks for 
which another way is available. This rationale lay behind 
the type of instructional innovation promoted among the 
intensive study programs. Some of the instruction in plaster 
application, traction hanging, interpretation of x-rays, and 
the appropriate use and handling of orthopaedic surgical 
instruments is possible by organizing the Study. This type 
of study substitutes for the instructor a set of materials 



*See Supplement for the folio of staff development 
materials. 

ic 'k 

See also p. 13 for discussion of the task forces 
related to this area. 
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guiding the resident through the more elementary aspects 
of the subjects and allows the resident to procede through 
the set at his own pace and whenever he has the time. The 
instructor's time is thus saved from repetitious instruc- 
tion for that instruction which only he can provide. 

1. Radiology Teaching Package . The Radiology Teach- 
ing Package was an outgrowth of the material devel- 
oped at the University of California at San Fran- 
cisco by Doctor Steven Ross, a radiologist. 

Dr. Ross developed teaching materials in response 
to a number of needs that he saw in the field of 
ratiology teaching. The first was that the resi- 
dents needed to be presented with radiological 
materials in the order of increasing difficulty 
so that in the learning process he was not encoun- 
tered by the wide variety of trauma that entered 
the clinical situation but was able to progress 
from the more easily recognized cases to the more 
difficult ones. Secondly, that the resident could 
study these materials on an independent and self- 
paced basis. As a result of the interest of the 
Orthopaedic Training Study in his work Dr. Ross 
has begun development, with the assistance of the 
Orthopaedic Department of the University of 
California, San Francisco, of several sets of 
orthopaedic resident radiology teaching packages. 
The Orthopaedic Training Study itself, to deter- 
mine the effectiveness of such a teaching method, 
has developed a four series set of materials which 
are designed to familiarize chiefs, attendings and 
residents with the concept, and provide a model 
for which they can develop their own teaching 
materials. The radiographic series we have devel- 
oped, illustrating the various types of pathology, 
are given anatomic location with attached atten- 
uating histories and obscured correct interpreta- 
tions and bibliography viewed by the trainee only 
after he has committed himself to an interpreta- 
tion of the films. Overlays are available for pre 
and post-testing and help to coordinate the de- 
scription of fine and obscure detail. 
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2. Plaster Skills . it was observed in many of the 
programs that the basic principles underlying 
the use of plaster were never systematically 
presented to the resident. It was expected that 
he would obtain this knowledge from one or another 
of the attending staff involved but no attending 
was individually responsible for teaching it. 
Plaster skills, being one of the most basic of 
orthopaedic psychomotor skills was selected first 
in our development of psychomotor innovations. 

The initial phase of the plaster lab was developed 
by one of the six month Fellows in Orthopaedic 
Education, who had just completed his residency 
training program. Under his direction an intensive 
investigation of the literature and rationale that 
had been developed for psychomotor training, which 
came mostly from industry, was investigated and a 
pilot unit was prepared. This unit was then cir- 
culated to the intensive study programs during 
site visits and comments and suggestions about 
both the methodology involved and its potential 
value to residency training programs was solicited. 
One of the curious phenomenon which resulted from 
our site visits was that there was a divergence of 
opinion between residents and attendings about 
plaster skills instruction. Residents saw a real 
need for the plaster laboratory early in their 
training, but attendings felt that the instruction 
was given to all residents at some time. The 
attendings, while they understood the rationale 
presented, did not feel that the laboratory was 
a necessary adjunct to their programs. Upon dis- 
cussion of the residents and attendings it became 
obvious that much of the instruction that the 
attendings took for granted as having been taught 
had been missed in many of the residents' training. 
Therefore, such a planned sequential program for 
every resident, seemed to the residents to be most 
desirable. After considerable discussion and lay- 
ing of the evidence, the attendings arrived at the 
same conclusion. This discussion and consensus 
occurred in about 80 percent of the programs in 
the second round of site visits. Subsequently, ■ 
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the total package of plaster skills .was developed 
and expanded to a series of eight lessons which 
was then field tested in three of the intensive 
study programs . Based on their comments to date, 
the materials have been revised and reviewed by 
the chiefs of the programs in the intensive study 
group for further development. 

3. Traction Laboratory . The second innovation to 
be developed was the traction laboratory. This 
innovation was developed by two residents, one 
of whom wrote the instructor's manual, while a 
second provided the ancillary (slides, tapes, etc.) 
material which were the. actual instructional tools 
for the laboratory. Again, this was tried out in 
two of the intensive study programs. Comments 

and suggestions were then used to further refine 
the laboratory. 

4. Psychomotor Skills - Surgical . Due to the perceived 
need for further development of the basic psycho- 
motor skills needed by beginning orthopaedic resi- 
dents, a task force was assembled to study the 

area of surgical technique and instrumentation which 
should be included in a basic surgical skills 
laboratory. After defining the instruments which 
would be considered, each of the participants 
selected an area to be developed both in the form 
of a sound slide presentation and a video tape of 
the dynamic aspects of instrument usage, rather 
than the static one, which could be demonstrated 
well on sound slides. It was decided to focus the 
laboratory around the use of particular instruments 
rather than around surgical procedures, both as a 
mechanism for the organization of individual study 
units independent of one another and for the simpli- 
fication in the presentation of the large amount of 
information, considering the large number of indi- 
vidual instruments which could be used in any 
procedure. The focus on instruments themselves 
avoided a duplication of instrument usage in the 
various operative procedures. The instruments 
which were selected for development in the first 
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meeting were as follows: scalpel, needle holders 

and needles (suturing) , hand saws, periosteal 
elevators, hand drills, drill bits, depth gauges, 
screw drivers, hemostats, rongeurs, wires, pin 
cutters, bending iron, wires and surgical pins, 
reamers, bone halters, rasps, osteotmee, curettes 
and gouges. 

4. Relationship of Input Training and Output Variables . 
The Study produced, in 1967, a data bank with the guarantee 
of anonymity for any individual resident, that allows an 
investigation access to a continuous record of resident pro- 
gress as it is assessed through annual In-Training Exami- 
nation (OITE) . Additionally, the data bank contains 
standardized perceptor ratings of resident achievement of 
knowledge, problem solving, and interpretive skills. Results 
of the Board Certifying Examination, which probes complex 
cognitive processes, professional behavior, and selected 
technical skills, are also part of the orthopaedic data 
bank. 



The model chosen for the study design was borrowed 
from the economics model of input-process-output. The 
elements of each are examined individually; for example, the 
characterization of the input is considered first, with the 
resident's background, attitude, fund of knowledge, etc., 
with which he comes to the field of orthopaedics, then the 
method of which this individual is treated or processed 
through a program for example, the settings the hospitals 
and various kinds of service rotations, etc. These are 
then related individually to the output measures. Finally, 
a variety of statistical techniques are useful in studying 
the interaction between various elements of the input and 
processing or treatment. 
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The analysis of the data was done at two levels. 

First, each of the many responses given by program chiefs, 
hospital administrators, attending staff, and residents, 
to the questionnaires were tabulated then summarized in 
the reports in Section II. Second, from the questionnaire 
results, hypotheses were generated at Advisory Committee 
Meetings, intensive study program chiefs' meetings, and 
personal observations of the Study Staff during site visits. 
These hypotheses stated expected relationships of charac- 
teristics of orthopaedic training program's facilities and 
organization, and characteristics and attitudes of residents 
and attendings to a criterion measure of resident 
performance. The criterion selected tipon which to judge 
the success of a resident's education was the OITE total 
multiple choice score and the total PMP score. The hypoth- 
eses tested in this report are by no means exhaustive of 
the areas in which investigation is needed nor are they, 
in many instances, refined enough for the results to be 
applicable to any particular program. Further evaluations 
of program efficiency and effectiveness need to be carried 
out in individual training programs to assure applicability 
of results. Hovevor, the results of the hypothesis test- 
ing research presented here is the first effort to investi- 
gate training programs. As such, its purpose is to stimu- 
late the orthopaedic and the educational research community 
to do further studies indicated by the trends and relation- 
ships uncovered by the present Study. 
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INTRODUCTION 



One of the first tasks confronting the Advisory Com- 
mittee and professional staff wa3 the selection of the Study 
samples. The objectives of the Study (see page 2 & 3) suggest 
that data collection from the introduction of innovations be 
divided among three board classifications of programs. 

1. Population : All programs would be asked to supply 

certain information, such as, program length, aims, 
and other characteristics (see Supplement for a 
list of all data collection instruments and by whom 
they were completed) . 

2. Survey Study S a mple ; A limited number of chiefs of 
programs (38) wore invited to participate in data 
collection efforts which would be more extensive 
than in the population at large (see Supplement) . 

Of those invited, 31 accepted the invitation to 
participate in the Study. The resulting group of 
programs was called the Survey Study Sample. 

3. Intensive Study Sample ; Sixteen program chiefs were 
invited to participate in an intensive study of 
their programs; all accepted. This group agreed to 
permit data gathering, not only about its program 
and institutions, but about residents and attendings 
as well. They also agreed to participate in the 
development and trial of innovative methods of in- 
struction which were developed by the Study staff. 

Both of the samples (2 and 3 above) were selected by the 
Advisory Committee in conjunction with the Study staff in order 
to reflect a similar ratio of a number of characteristics 
believed to influence the quality of orthopaedic education as 
would be found in the total resident training program population 

It was intended to provide two samples that would reflect, 
among others, the variations in program size, geographic 
location, institution type (university, university-affiliated, 
or independent), numbers and types of affiliated institutions, 
program mission, patient composition, and program length sim- 
ilar to those found in the total group or population. 
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The extent to which programs cooperated can be seen 
from the rate of return for the various data collection 
instruments (see Supplement) . 
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In the Final Report of the Phase I of the Orthopaedic 
Training Study, four areas were suggested for a description 
of program variation in orthopaedic training: program 

organization, program objectives, program operation, and 
program evaluation. These areas will serve as a basis for 
the following description of the nature and variation of 
orthopaedic training programs presently existing in the 
United States. In addition, ar attempt will be made to char- 
acterize a typical university, university-affiliated, or 
independent program. 

The same Program Questionnaire . (see Supplement) was 
administered twice to directors of all the 187 approved 
orthopaedic residency programs. There were 150 responses in 
1969 and 131 in 1971. In addition, there were 116 responses 
to the Institutional Description Form from approximately 200 
institutions associated with the 51 programs included in the 
Orthopaedic Training Study sample. The Program Questionnaires 
will be the main source for information in this section of the 
report. However, the information from the 131 Program Question- 
naires returned from the 1971 administration will be reported 
in this analysis only when they reveal apparent differences 
between the earlier 1969 results and the 1971 results. 

In the main, however, there did not seem to be much 
change in orthopaedic training from 1969 to the present time 
as reported by these questionnaires. This is not particularly 
surprising as institutional change traditionally occurs slowly. 
It is important to note, at this juncture, that where changes 
are indi> ated, the interpretation of these results must be 
speculative in nature. Nineteen fewer programs responded to 
the administration of the Program Questionnaire in 1971, and 
it may be that the apparent changing results are a function of 
non-response rather than of actual sample differences. With 
this reservation, where changes are apparent, they will be 
indicated in this analysis. 

PROGRAM ORGANIZATION 

This section of the report deals with the personnel, fa- 
cilities, and resources of the Orthopaedic Training Programs 
and their associated institutions in the sample. Unless other- 
wise specified, the statistics presented in this section of the 
report refer to all programs in the Study sample. 
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It should be noted that this section of the report is 
concerned with a description of a typical orthopaedic pro- 
gram, without regard to program type differences. The final 
pages of this section detail the differences among types of 
orthopaedic programs in the hopes of further highlighting 
variations in residency programs. 

GENERAL PROGRAM CHARACTERISTICS 



Approximately 40 percent of all approved programs from 
the 1969 sample and 47 percent from the 1971 sample identify 
themselves as university programs.-*- Approximately 29 per- 
cent of the programs identify themselves as affiliated with 
either a medical school or a university, ^ while 29 percent of 
the programs in 1969 and 23 percent in 1971 identify them- 
selves as independent programs. ^ (it should be noted that 
these percentages are based upon the description of the pro- 
gram type by the chief of the program.) It is somewhat more 
common for university programs to be organized as a division 
within the department of surgery than as a separate depart- 
ment; while independent programs are more likely to identify 
themselves as departments of orthopaedic surgery in non-uni- 
versity hospitals. In 70 percent of all the programs, the 
chief has academic rank, though not necessarily tenure. This 
indicates that almost all of the chiefs of university and 
affiliated programs have academic appointments. Sixty per- 
cent of the chiefs have offices located in the principal 



Typically, university programs are those orthopaedic pro- 
grams which are an integral part of a university with the 
attending staff having university appointments. 

2 

Typically, university-affiliated programs are those 
orthopaedic programs which are connected to a university or 
medical school through a variety of formal arrangements. The 
attending staff of these programs may hold university 
appointments . 

3 

Typically, independent programs are those orthopaedic 
programs which have no formal connections with a university 
nor are their staffs drawn from university faculties. 
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ins titut. ion of the program. In slightly over one-third of 
the programs, the bulk of the chief's income derives from 
salary; while in an almost equal number, the chief receives 
virtually no income from that source. 

Eighty percent of the programs embrace several asso- 
ciated institutions usually within the immediate geographic 
area. These associated facilities are likely to include a 
private community hospital (in 70% of programs reporting), a 
"crippled children's" hospital (in 62%), a Veterans' Admin- 
istration Hospital (in 44% of the cases in 1969 and 50% in 
1971) , and a "charity" hospital (in 43%) . 

Even in these multi-institutional programs, the chief 
of the program often retains direct control over all aspects 
of the program. However , in approximately half of the pro- 
grams, some of the control is delegated to chiefs of service 
of constituent institutions, and in approximately one- fourth 
of the reporting programs, the chief of service is reported 
to be completely independent of the program chief. Similarly, 
even though training is distributed through several institu- 
tions, the resident is likely to encounter substantially the 
same attending staff in all associated institutions; only 
slightly more than one-third of the programs report a sub- 
stantially different staff in each associated institution. 

With respect to the general characteristics of the asso- 
ciated institutions, almost one-third identify themselves as 
private hospitals. ^ Another third of the institutions are 
about evenly divided in identifying themselves as university 
hospitals and public hospitals of the city-county type. The 
balance of the institutions report being hospitals which rep- 
resent some combination of the preceding, and by military, 

VA, and other non-profit organizational hospitals. 

The board of directors is wholly appointed for over 40 
percent of these institutions. In less than 10 percent of 



^One hundred sixteen institutions are included in this 
analysis, which constitutes approximately 50 percent of the 
institutions associated with programs in the study sample. 

The Institutional Description Form was administered only once, 
in 1969. 
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the cases, the boards include members elected from the commu- 
nity-at-large. Almost three-fourths of the institutions employ 
a professional hospital administrator; most of the rest have 
a physician administrator. The medical staff administration 
is elected from the staff-at-large in 37 percent, and is lim- 
ited to chiefs of services in 34 percent of the institutions 0 

Most of the institutions associated with programs in the 
study sample rely on multiple sources of support. Indeed, 
only four percent of the institutions obtain total support 
from private foundations and only 10 percent report that they 
receive n<D government funds for any purpose. Over one-third 
of the institutions report that more than 50 percent of their 
support is obtained from governmental sources (federal, state, 
and local) , while almost half report that over 50 percent of 
their budget is derived from patient fees. Four percent of 
the institutions report that over 50 percent of their support 
comes from university budgets. 

PERSONNEL 



While some programs have fewer than five residents and 
six attendings and others more than 60 and 80 respectively, 
most reported between six and 12 residents in 1969. These 
numbers increased to between seven and 16 residents in 1971. 

In 1969, the bulk of the programs reported having between five 
and 20 attending staff members, while the numbers increased in 
1971 to between seven and 21. In the "typical" program in 1969 
there were eight residents, 11 attendings, and no research or 
clinical fellows, while in 1971, the "typical" program was com- 
prised of 11 residents and 12 attendings. This increase in 
number of residents in a typical program may be attributable 
to pressure on the medical community to develop more trained 
physicians as well as the increasing attractiveness of the 
specialty of orthopaedics. 



Only 20 percent of the programs reported any fellowship 
staff and no program reported more than four on the fellowship 
staff . 
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In addition to a full range of medical specialists 
available as professional consultants in virtually all pro- 
grams, the subspecialties most often represented by the 
orthopaedic attending staff are hand surgery, orthopaedic 
pathology, pediatric orthopaedics, spine, and rehabilitation. 
Table 1 presents information concerning the percentage of 
programs reporting at least one subspecialist in their pro- 
gram within a particular category. The table should be 
interpreted in terms of programs indicating the presence of 
subspecialists, not in terms of numbers of subspecialists 
within a particular program. 

TABLE II - 1 

PROGRAMS REPORTING SUBSPECIALISTS 
(Percentages) 



1969 1971 



Rehabilitation 
Bio-mechanics 
Orthopaedic Pathology 
Hand Surgery 
Neurologic disorders 
Pediatric Orthopaedics 
Spine 

Other subspecialists 
None 



49.0 


53.0 


20.0 


32.0 


59.0 


63.0 


76.0 


84.0 


30.0 


31.0 


59.0 


73.0 


53.0 


58.0 


15.0 


9.0 


9.0 


5.0 



Table 1 indicates that the trend toward specialization 
within the general field of medicine is also evidenced within 
the field of orthopaedics. This is interesting, since this 
specialization trend is noticeable within a space of only two 
years. Especially noteworthy is the increase in programs 
reporting bio-mechanics and pediatric orthopaedics as sub- 
disciplines represented by specialists on the attending staffs. 



In addition to the subspecialists represented on the 
attending staff, these programs report that allied health per- 
sonnel afford the programs additional specialties. Ninety-nine 
percent of the programs report physical therapists comprising 
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part of their allied health staff, while 95 percent of the 
programs have social workers on the ancillary staff. Over 
80 percent of the programs report prosthetists, orthotists, 
and occupational therapists as comprising part of their 
allied health personnel. Additionally, over 80 percent of 
the programs report psychiatrists as part of the program 
personnel - 

Other supporting services within the institutions asso- 
ciated with the sample progreims are most likely to include a 
nursing supervisor for in-patient services (in 98% of the 
institutions reporting), for the operating suite (in 97% of 
the institutions), for the emergency room (in 65% of the 
institutions), and for the out-patient service (in 60% of the 
institutions. An administrative supervisor is slightly less 
frequently available. For both in-patient and out-patient 
services, the administrative supervisor is available in 80 
percent of the institutions. This percentage drops to 60 for 
the operating room and 50 percent for the emergency room. 

Eighty percent of the out-patient clinics have a registered 
nurse, but less than half have practical nurses, orderlies, 
plaster technicians or social workers, assigned solely to the 
orthopaedic clinic. Additionally, one is least likely to find 
psychological services (available in 43% of the institutions), 
counseling services for vocational rehabilitation (in 39% of 
the cases), and home nursing services (in 41% of the institu- 
tions) a s part of the support services within the institutions 
associated with the xerograms in the Study sample. 

FACILITIES 

In addition to both the general orthopaedic and children's 
clinics found in virtually all programs, most programs report 
the following out-patient subspecialty clinics as also avail- 
able in at least one institution associated with the program: 
fracture (in 87% of the reporting programs), amputation and 
prosthetics (in 77% of the programs), arthritis (in 60% of the 
programs), and scoliosis (in 52% of the 1969 responding pro- 
grams and in 63% of the 1971 responding programs) . However, 
only 40 percent of the programs report a long-term follow-up 
clinic where patients treated five years in the past are 
brought back for review. 
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With respect to the institutions associated with pro- 
grams in the Study sample, the types of facilities most 
commonly included as part of the institutional structure are 
the following: 

Out-Patient Services ; About one -half of the institu- 
tions associated with the Study programs report orthopaedic 
subspecialty clinics. The most frequently encountered clinic 
was a hand clinic, with 72 percent of the institutions report- 
ing this subspecialty. The presence of amputation prosthetics 
and children's subspecialty clinics were reported by 70 and 65 
percent of the institutions respectively, while a special 
arthritis clinic was reported by about one-half of the 
institutions . 

In-Patient Services : The institutions associated with 

the study programs reported that the average number of beds 
available to all services is 511, and to the orthopaedic 
service, 64. ^ While 44 percent of the institutions report that 
they have no private, single-bed rooms, such facilities account 
for over one-half of the orthopaedic beds in 40 percent of the 
institutions . 

Seventy-six percent of the institutions report in-patients 
with fracture problems; almost two-thirds report in-patients 
with other trauma problems and hand problems. However, only 
slightly more than one-half report in-patients with rehabili- 
tation and arthritic problems. 

In approximately one-third of the institutions, admission 
for both private and non-private patients are scheduled accord- 
ing to general bed availability. Only three percent of the 
institutions report that patients are screened for admission 
strictly according to the teaching value of their problems, 
though an additional 21 percent report less strict application 
of this screening criterion. In 58 percent of the reporting 
institutions, residents do not have authority to arrange for 
admissions . 
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To the extent that respondents failed to answer all 
questions, these figures may represent an underestimate. 
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The mean autopsy rate for these institutions is 48 per- 
cent and that for the orthopaedic services is 43 percent. 





Surgical Facilities : The mean number of operating rooms 

in a sample institution is eiglrt. The mean number of rooms 
assigned specifically to orthopaedics is one. In 67 percent 
of the reporting institutions, the individual surgeon sched- 
ules his own elective cases. In 84 percent, he schedules his 
emergency cases, while in nine percent of the institutions, 
scheduling of emergency cases is done through a chief surgical 
officer. In 63 percent of the institutions, orthopaedics is 
assigned specific operating days. The operating room is under 
hospital control in half of these institutions, general surgery 
control in 22 percent and individual specialty control in 12 
percent of the reporting institutions. Fifteen percent of the 
institutions assign nursing and technical personnel according 
to general availability. Approximately three-fourths of the 
institutions report having an adequate assortment of surgical 
equipment and appliances readily available, though 11 percent 
require special purchase of endoprosthetic devices and three 
percent require that the surgeon supply his own instruments . 

All institutions have x-ray facilities available in the oper- 
ating room, while 23 percent have this type of equipment and 
technicians assigned specifically to the operating room 
routinely. in 53 percent of the institutions, anesthesiologists 
administer all general anesthesia. Only nine percent of the 
institutions have nurse anesthetists who work without super- 
vision of an anesthesiologist. Finally, in 73 percent of the 
institutions, both the surgical and anesthesia services admin- 
ister regional anesthetics. 

Supporting Services ; Satisfaction with the quality of 
supporting facilities was most likely to be reported for med- 
ical and nursing administration (in approximately 90% of the 
institutions), and for x-ray, special laboratory, prosthetics 
and physical therapy facilities (in 80 to 85% of the 
institutions). Social service, occupational therapy and 
orthotics were viewed as adequate or superior in 60 to 70 per- 
cent of the institutions: while research, school facilities 

for patients, convalescent, psychiatric counseling, and voca- 
tional rehabilitation were most often reported as unavailable 
or inadequate by 40 to 50 percent of the institutions. 
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Although approximately one-third o£ the institutions 
have formal arrangements or affiliation with convalescent 
and custodial care facilities, the majority do not have 
such arrangements. In fact, at least three percent of the 
institutions are in communities where no such facilities 
exist . 



Record Facilities : Over two-thirds of the reporting 

institutions have the capability of retrieving in-patient 
charts according to diagnosis or treatment, whereas only 
one-third can retrieve out-patient records in this manner. 

Library Facilities : Whereas 80 percent of the insti- 

tutions have a medical school library available, only about 
one-half have an orthopaedic departmental library available 
to the residents for at least eight hours daily. More than 
half of the libraries used by the residents are relatively 
small, fewer than one-sixth reporting collections of over 
75,000 volumes. 

Secretarial Service ; Approximately three-fourths of 
institutions report that such services are available to 
house staff for in-patient records and activities, while 
two-thirds report such facilities available for out-patient 
activities. One-quarter of the institutions report clerical 
support for research endeavors. Almost one-third report lia 
son personnel for community resources. Less than 10 percent 
report no clerical support available. 

RESOURCES 



One of the vital resources of any orthopaedic training 
program is the patient population. In this section of the 
report, we will attempt to summarize the variations in pa- 
tient population and the resultant implications of this vari 
ation for residency programs. 

Size of Patient Population ; The average number of out- 
patient visits to orthopaedic clinics is 3,858 although the 
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number varies significantly from one institution to another 
within a program.”^ 

in addition to the orthopaedic clinic, about half the 
institutions include a separate arthritis clinic in which 
the main number of annual out-patient visits is 454. Finally, 
the average number of annual out-patient visits in the medical 
and surgical clinics of responding institutions is 17,562 and 
15,767 respectively. 

Clinical Variety : Despite differences among institutions 

in the pattern of administrative responsibility for patient 
care, a few widely applicable generalizations can be made. 
Approximately 80 to 90 percent of the institutions report that 
the orthopaedic department assumes administrative control over 
patients with the following problems: fractures, musculo- 

skeletal tumors and amputations occasioned by other than pe- 
ripheral vascular disease. It is interesting to note that in 
approximately one-fourth of the institutions, the orthopaedic 
department also assumes administrative responsibility for 
patients with lumbar disc, hand problems, chronic spinal cord 
injuries, or rehabilitative cases requiring long-term care. 
Finally, in between 10 and 15 percent of the institutions, 
the orthopaedic department assumes administrative responsi- 
bility for patients with arthritis, cervical disc, acute 
spinal cord injuries, or for patients requiring amputation 
due to peripheral vascular disease. 

Most programs report that the most frequently encountered 
problem is trauma, representing up to 60 percent of all 
patient problems in a few programs. Rehabilitation problems 
are least common, representing less than 20 percent of all 
patient problems in most programs. 

Age Distribution : In hospital settings other than those 

limited to specified groups (military, veterans, or children), 
the most frequent pattern with respect to the age distribution 
of the patient population is that 80 percent are adults and 
20 percent are children. This picture is characteristic of 



7 

To the extent that respondents failed to answer all 
questions, these figures may represent an underestimate. 
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over 40 percent of cill institutions to which residents are 
assigned. 

Resident ial Distribution : Over one-third of the insti- 

tutions in the Study sample serve patients who reside in 
metropolitan areas of over one million persons. Fewer than 
10 percent of these institutions serve predominantly rural 
populations of under 10,000 people, while most of the 
remaining serve patients from urban centers of between 
100,000 to 500,000 people. 

Soclo -F conomic Status ; Although the ratio between pri- 
vate and non-private patients varies significantly among 
programs, only eight percent of the programs report no "pri- 
vate" patients available in the entire program, and only 
one percent report no "non-private" patients.® Secondly, 
whether the private or the non-private patient predominates 
in a parti cular program, that type usually constitutes no 
more than 80 percent of the total patient population. In 
short, it would appear that most programs serve patients from 
various socio-economic levels. However, the opportunity for 
an individual resident within a program to gain experience 
with patients of various socio-economic levels is dependent 
upon the specific rotations available. Thirty-seven percent 
of the institutions report that they deal predominantly with 
patients in the lower socio-economic groups.® Indeed, over 
a fifth of all institutions report that they serve predomi- 
nantly indigent populations and another seven percent serve 
predominantly Medicare patients. Of the remainder, 51 per- 
cent of all institutions report that more than half of their 



O 

Private patients are, in general, patients who are 
billed directly by their attending physician for services 
rendered. Non-private patients are, in general, patients 
who are not billed directly by the responsible physician. 

9 

This category was not defined in the Institutional 
Description Form, and hence, it is not possible to state 
the income boundaries of thi- designation. 
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private patients have care financed through insurance pirns, 
and only four percent of the institutions report that between 
20 to 60 percent of these patients finance their care wholly 
from private resources. 

In addition to this economic diversity in the patient 
population, 60 percent of the institutions report that they 
serve non-English speaking patients to some degree. However, 
there is an interpreter available in slightly more than one- 
half of those institutions reporting a substantial number of 
non-English speaking patients. 

Approximately one-half of the institutions report dint 
there is a communications problem between patients and 
residents. This may, in part, reflect the disparity between 
the socio-economic and ethnic backgrounds of the patient 
population and that of the resident staff. 

It is clear from the foregoing that the kinds of clinical 
problems and the varieties of socio-economic groups to which 
the resident is exposed depends almost entirely on the nature 
and location of the institution in which he. is trained. Loss 
than onc-fourth of the institutions report that they do 
screening for the educational value of the medical problem. 

PROGRAM OBJECTIVES 

Orthopaedic training programs have both medical and 
educational objectives which are said to direct the function- 
ing of the program operations. Eighty-four percent of the 
programs report that their primary objective is to "produce 
clinicians oriented to community practice". The production 
of subspecialists or academicians is ranked as the primary 
objective by only 15 percent of the programs. The production 
of research-oriented orthopaedists is regarded as of first 
importance by only one percent of the programs, while slightly 
over one-half rank this objective as third in priority. 

The most frequently encountered educational pattern 
characteristic of almost one-half of all programs, consists 
in four years of orthopaedics following internship, although 
approximately one- third of the programs offer a three year 
orthopaedic residency following 12 months of general surgery. 
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Nrarly half the programs offer complete training in ortho- 
paedics plus options for development of more specialised 

interests. Virtually, nil others offer complete training 

*10 

in orthopaedics as standardly defined. 

Two-thirds of the programs include a rotation in neuro- 
surgery sometime during the four years and over one-half 
include rotation in plastic surgery. The sample from I960 
reported that, in 53 percent of the programs, a rotation in 
"trauma" was also offered, while the 1971 sample reported 
this rotation available in 65 percent of the programs. 

Typically, clinical teaching includes the following 
types of . ''tiviti.es on a regularly scheduled basis: didactic 

instruction, bedside teaching rounds, group conferences, wo rk 
rounds, surgical experience and out-patient and specialty 
clinics supervised by an attending. In 77 percent of the 
programs, basic science instruction is offered within the 
program and this instruction is distributed throughout the 
four years 1 .. in approximately one— third of the reporting pro- 
grams in .1.969, the resident was required to submit a research 
paper, and in all but a few of those programs, ho is expected 
to carry on his investigation and clinical work concurrently. 

In 1971, the percentage of programs requiring a research 
paper increased to 45 percent. 

In order to determine if this apparent increase in empha- 
sis on research was an artifact of the differences in samples 
between the two years, a re-analysis of the results were under- 
taken using only programs which had responded to both of the 
quest ionnai.ro administrations. The 117 institutions which 
responded to both the 1969 and 1971 questionnaires showed an 
increase in emphasis on research. Thirty-six percent of these 
programs reported that a research paper was mandatory in 1969, 
while that figure is 42 percent in 1971. However, despite the 
increased emphasis on the submission of a research paper, there 
is not a corresponding increase in the percentage of programs 
allotting specific time for work on the paper. 



* ^Standardly defined training involves complying with 
the requirements of the Board. 
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In about two-thirds of the programs reporting in 19G9 
and 78 percent reporting in 197a, presentation of papers 
at national meetings is actively encourac .. These per- 
centages were identical when the 1.17 programs responding 
to the 1969 and 1971 questionnaire administration were 
analysed . 

Attendance at one (or more) annual national meeting 
was mandatory in 53 percent of tne programs in 1969 and in 
61 percent of the programs reporting in 1971. 

There appears, then, to be a trend toward increased 
resident participation in research activities in residency 
programs, even over the short time of two years. This 
trend may have implications for objectives, further program 
directions, and allocations of resources within the ortho- 
paedic community. 



PROGRAM OPERATION AND EVALUATION 

The variations in program operations are undoubtedly a 
function of the organisation and objectives of the program. 

7iS a consequence, this section of the report connects the 
previous material with information about resident responsi- 
bility, in terms of teaching and patient care, and with feed- 
back and supervision mechanisms. 

Resident Teaching Responsibilities : in most programs, 

the resident is expected to participate in the instruction 
not only of other residents and interns, but also of medical 
students (in 60% of the programs reporting in 1969 and in 67% 
reporting in 1371) , and of allied health professionals includ- 
ing nurses and physical therapists. 

Res i dent Patient Care Responsibilities : Table 2 presents 

data il lust ratine/ the responsibility of residents for patient 
care over the length of the residency. Eighty-eight percent 
of the program chiefs report that residents begin performing 
simple orthopaedic operations in the first year of training 
and progress systematically toward the more complex procedures 
wiuh increments in experience. It is clear, however, from 
Table 2 that that first-year residents in some programs are 
expected to assume a level of responsibility that fourth-year 
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residents in other programs never assume. Thus, for example, 
while residents in most programs can expect to assume full 
medical responsibility for frequently encountered types of 
surgical trauma sometime between late in their second-year and 
early in their fourth-year, three percent of the chiefs 
report that such responsibility is usually assumed by first- 
year residents and five percent of the chiefs report that 
such responsibility is never assumed by residents. 

It is evident, then, that there is a rate differential 
among programs in the level of responsibility allocated to 
residents. 

It is clear, also, from Table 2 that there has been an 
increase in providing residents with more responsibility 
earlier in their residency from 1969 to 1971. There are sub- 
stantial increases in resident responsibility displayed on 
this table, such os , the increase from 39 percent in 1969 to 
59 percent in 1971 of programs reporting thac first-year resi- 
dents are responsible for out-patient trauma cases. This 
increase could not be accounted for solely by idiosyncrasies 
of the responding programs in the 1971 sample. In every case, 
without reversal, there was an increase in responsibility 
assumed by residents at each type of orthopaedic procedure. 



SUPERVISION AND FEEDBACK 



in approximately two-thirds of the programs, chiefs 
report that actual supervision of residents in the operating 
room is provided in accord with the needs of the individual 
resident. in the rest of the program, actual supervision in 
surgery is provided for all cases regardless of complexity. 
Finally, 89 percent of the chiefs report that residents are 
regularly advised of their rate of progress and their areas 
of deficiency. 

SUMMARY 

The preceding section has attempted to describe ortho- 
paedic training programs with major emphasis on the nature 
and variation of these programs in terms of facilities, per- 
sonnel, organization, and resources. Less information was 
presented concerning the objectives, feedback mechanisms, and 
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evaluativc procedures employed by the programs. It would 
seem that future studies along these lines might be most 
productive if those relatively unexplored aspects of train- 
ing programs were thoroughly investigated. 

A PROGRAM TYPOLOGY 

In order to more clearly depict the nature and varia- 
tion of orthopaedic residency programs, a typology of pro- 
grams was developed based on the affiliation dimension of 
program organization. That is, a characterization of a 
typical university, university-affiliated, and independent 
program was created. Table 3 displays that characterization. 

The university programs appear* to have more available 
facilities and resources than the other two types of programs. 
There are more out-patient clinics, subspecialists, and asso- 
ciated hospital facilities available to university programs 
than university-affiliated and independent programs. Uni- 
versity programs also tend to supervise resident training more 
closely and wait longer to give residents patient care respon- 
sibility for simple orthopaedic procedures. it is also a 
characteristic of university programs that residents are per- 
mitted to become involved in special interest areas and are 
encouraged to give papers at annual meetings . 

University programs tend to be more similar to university- 
affiliated porgrams than are either of these types to indepen- 
dent programs. These differences are particularly apparent 
with respect to general organization and administration. These 
differences tend to disappear with respect to resident training 
per se. 



Independent programs are characterized by more authority 
vested in the chief of the program than chief of staff, by an 
emphasis on internship performance as a criterion for admission 
to the program, and by fewer additionally required rotations 
for residents. Additionally, independent programs have fewer 
supspecialists, associated facilities, and out-patient clinics 
than university or university— af filiated programs. Independent 
programs permit residents to assume non-surgical patient care 
responsibilities early in their residency but maintain closer 
supervision for surgery at all levels of competency. Finally, 
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V 



these programs are characterized by comparatively more 
emphasis on attendance at annual meetings and optional 
research papers, and with higher pay for the resident staff. 

It should be noted that, in general, the characteriza- 
tion of the three types of programs are quite similar. There 
are standard facilities and resources available irrespective 
of program type, admissions criteria which are accepted in 
all three types of programs, and educational objectives and 
mechanisms for accomplishing these objectives which appear 
to remain similar throughout orthopaedic residency programs. 
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II - nature and variation 

B - RESIDENTS AND ATTENDINGS 
The Typical Orthopaedic Resident 

The typical orthopaedic resident identified by the 
Study is 30 years old, male, married, Caucasian, has one 
or two children, comes from the Midwest or Middle Atlan- 
tic states, frequently has a father or uncle who is an M.D., 
and often a wife, sister, or mother in a health-related 
occupation, usually nursing. His educational background and 
his family's economic status are well above average. One- 
third of the residents have fathers with a professional de- 
gree and one-fourth have mothers with a college degree; 

75 percent have fathers in executive or professional posi- 
tions; 52 percent have fathers who are self-employed. The 
resident is among the intellectual elite; nearly half of 
the resident population in the Study sample were in the 
upper five percent of their high school class; over two- 
thirds were in the upper quarter of their college class; and 
nearly half were in the upper quarter of their medical school 
class . 



A desire for independence, an interest in science, in 
people, and in the workings of the body, and the prestige 
of the profession were strong influences in his selection 
of medicine as a profession--a decision he probably made 
about the time of high school graduation. Of little impor- 
tance were books or articles, movies about medicine, or 
previous vocational guidance. The nature of the specialty 
and of the clinical material, a desire to perform surgery, 
an ability to work with his hands, and his experiences dur- 
ing the internship were strong determinants in his choice 
of orthopaedics as a career — a decision he tended to delay 
until his internship year. 

In general, reputation of the institution and its geo- 
graphical location, as well as financial considerations, 
played a major role in his choice of both a medical school 
and a residency program. However, in selecting the latter, 
he was also influenced by his desire to work with a particu- 
lar person and by his associations or experiences in medical 
school. Over 80 percent of the residents report that they 
were appointed to the program of their first choice; only 
five percent report that they were appointed to a program 
of third choice or lower. Approximately 60 percent of 
residents indicated at the time of the survey that they aspire 
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to private practice in a small orthopaedic clinic and that 
they hope to have some medical school affiliation and do 
some teaching. 

In order to gain further insight into the variations 
of the background of residents, the following description 
of an atypical resident is presented. All of the character- 
istics mentioned do not necessarily describe any one indi- 
vidual or set of individuals, but they exist in the popula- 
tion under study. These characteristics highlight the range 
of responses rather than the mean or average response on 
which the typical resident description is based. 

The Atypical Orthopaedic Resident 

The atypical orthopaedic resident could be 25 years 
old (the age of most senior medical students) or 37 years 
old in his sixth year of training (the age of two chiefs 
of orthopaedic programs) . He could soon be among the more 
than 60 residents who left orthopaedics or changed to an- 
other program during the last five years, more than half 
of whom were dismissed for irresponsibility (moral or 
ethical) or poor clinical judgment. His* father may not 
have completed high school, as was the case with almost 
20 percent of the residents, while his parental family 
income last year was probably less than $7,500, and his 
family's economic position last year was lower than it was 
10 years ago. Seven percent of the residents have accumu- 
lated over $10,000 in educational debts while one percent 
earned less than $4,000 as a senior resident. The atypical 
resident may be among the three percent who maintained a 
straight C average through high school, college, and medi- 
cal school, and did not take any biology courses in high 
school (5%) or college (1%) , still he was able to become 
an orthopaedic resident (even though he was admitted to a 
program which was his third or lower choice (5%). He 
could have decided to study medicine at age 27 and ortho- 
paedics at age 34. He could have elected medicine at age 
six and decided on orthopaedics as a college freshman at 
age 18. A majority have seriously considered another pro- 
fession. Consequently, our atypical resident might have 
become an educator (7%) / business administrator or lawyer 
(each 4%), forester or aeronautical engineer (each 3%) or math- 
ematician ( 2 %) . He may be among the 29 percent who read 
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medical literature for less than 10 hours a week and spend 
more time than that in employment outside the residency 
(12%) . In fact, almost one-third of the residents reported 
that they do work outside the residency in an area directly 
related to medicine, while this practice is expressly for- 
bidden by most training programs. 



Resident Attitudes 

As shown in tables 7 and 8, between 70 and 85 percent 
of the residents whose attitudes were surveyed showed a 
satisfaction with many aspects of their residency program 
and with the staff and residents with whom they work. They 
do, however, have some complaints about hospital procedures.** 
Their responses can be summarized in three categories, as 
f ol lows : 

1. Satisfaction with attendings as teachers. 

(See table 4) . 

2. Acceptance of responsibility to teach students 
and the satisfaction with the personal benefits 
of having to teach (improving technique with 
the extra practice and preparation by extra 
reading) . (See table 5) . 

3. Dissatisfaction with hospital procedures 
which waste resident's time ("scut work", 
excessive night duty, services which are 
inadequate or available only with much dif- 
ficulty).** (See table 5). 



♦♦Residents are equally divided on this issue, i.e., as 
many are satisfied as are dissatisfied. 

€ 
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TABLE II - 4 

{ 

Satisfaction with Attendings As Teachers 

1. The full-time faculty in my department understands 
and appreciates the contributions of the resident. 

2. I have sufficient opportunity to work closely with the 
senior (attending) staff. 

3. The attendings in my department exert a strong effort 
toward making the resident's experience a valuable one. 

4. My department is generally regarded as having a 
"stimulating" teaching staff. 

5. It was known to the residents that this residency 
program had been organized with specific objectives 
in mind. 

6. The full-time staff in my department places great 
emphasis on the practical management of problems. 

7. Attendings consciously strive to improve their per- 
formance as teachers. 

8. Residents are helped to understand the source of 
important problems they may be facing. 

9. when I began my assignment on this specialty, the 
resident's role was made clear to me. 

10. Constructive suggestions are offered to residents in 
dealing with their major problems. 

11. Evaluation of the residents goes on constantly. 

12. If I had it to do over again, I would select this 
residency program. 

13. The different skills found in the attendings are fully 
utilized . 

14. Most of what I have learned thus far has been super- 
vised to my satisfaction. 

15. The attending staff is sufficiently aware of my per- 
formance to render an accurate evaluation of my 
work. 

i 
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TABLE jl - 5 

r 

Resident. Teaching Responsibilities 

1. Students on my service do not hinder the performance 
of my duties. 

2. Having students around causes me to "read up" more 
than I might have done otherwise. 

3. Having to teach does not take time away from more 
important things I want to do. 

4. The presence of students on my service makes it a 
better " teach: ng service." 

5. I now believe this residency would be worse without 
students . 

6. Teaching students how to perform certain procedures 
helps to improve my own technique. 

7. Residents play a major role in teaching. 

8. The presence of students probably affects my learning 
one way or the other. 

9. My service is student- oriented . 



t 
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TABLE II - 6 

D.i i^r.n ' ^ r action wjth H o spital Procedures 



1. Tine spent arranging to have laboratory tests done 
detracts from time I could use on come more produc- 
tive areas. 

2. Residents do too many menial tashs on my service. 

3. Residents are exploited by the hospital and its 
attending physicians. 

4. I feel that the resident's assignments are determined 
primarily by a need to provide "service" to patients. 

5. Paramedical personnel have too much control over a 
resident'e activities. 

6. I have too much night duty. 

7. Clinical laboratory services are inadequate and 
available with too much difficulty. 

8. Demands on the residents are so great they arc al- 
most impossible to meet. 
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TABLE II - 7 



Many of the residents who responded to the survey 
AGREED with the following statements: 

» 

Eighty-five Percent Agreement : 

Regularly scheduled resident meetings are a valuable 
educational activity. 

Most residents here make an effort to keep up with 
the current medical literature. 

Residents are open and free about exchanging information. 
The residents get along together socially. 

Seve nt y Percent Agree ment : 

Residents play a major role in teaching on my service. 

I have sufficient opportunity to work closely with 
the senior (attending) staff. 

The attend i.ngs in my department exert a strong effort 
toward making the resident's experience a valuable 
one . 

Residents have the feeling that their work is an impor- 
tant activity. 

The full-time staff in my department places great 
emphasis on the practical management of problems. 

There is a congenial relationship between most of 
the attendings and residents on my service. 

Teaching students how to perform certain procedures 
helps to improve my own technique. 

Constructive suggestions are offered to residents in 
dealing with their major problems. 

Evaluation of the residents goes on constantly. 
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TABLE II - 7 (Cont'd) 



Residents are willing to help out when a fellow 
resident has a great deal of work to do. 

I have sufficient opportunity to develop and use my 
clinical skills during residency. 

If I had it to do over again, I would select this 
residency program. 






-49- 




TABLE II - 8 



More than 70 percent of the residents who responded 
to the survey DISAGREED with the following statements: 

Evaluation of a resident is based primarily on 
surgical skill. 

Students on my service hinder the performance of 
my duties. 

Residents do too much research. 

The morale of the resident staff is low. 

The number of private patients on my service is 
inadequate . 

Having to teach takes time away from more important 
things I want to do. 

I now believe this residency would be better 
without students. 

Instruction from the full-time faculty is too 
theoretical or abstract. 

Most of what I have learned so far in this resi- 
dency has been from paramedical personnel. 
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Residents' Work Week 

For seven days each resident kept a daily log of his 
activities in each of seventeen categories (of which the 11 
most prominent are included here) . These data were then 
combined to describe an average work week for residents in 
each of the four residency years. The findings are summa- 
rized in table 9. 

The most striking item is the length of the average 
working day, 11 to 12 hours, seven days a week over a four- 
year period. With such a pattern established during train- 
ing, the continued stamina and drive which seem to charac- 
terize the orthopaedic community should come as no surprise. 
Whether such intensity provides an optimal experience for 
learning other attitudes and values, as well as the problem- 
solving skills and reflective thinking which represent resi- 
dency program goals, is in question. However, as the third 
category of resident attitudes indicates, residents are 
equally divided regarding complaints about the amount of 
time spent in night duty, obtaining equipment and services, 
etc . 



The essentials of an approved residency of the American 
Board of Orthopaedic Surgery, like the essentials of an ap- 
proved internship, state unequivocally that these training 
experiences must first be designed for learning. In the 
course of this learning, medical services must be rendered, 
but service is relatively a secondary, rather than a pri- 
mary, outcome. While there may be differences of view about 
the extent to which on-the-job training is primarily educa- 
tional, there would probably be fewer disagreements with .the 
principle that effective learning requires some designed 
sequence, and that learning critical judgment in patient 
management demands graded and ever-increasing responsibility. 
The 11 time components reported in table II-9 (p. 52) are di- 
vided into three elements arbitrarily labelled formal l earn- 
ing (observing surgery, assisting surgery, attending learning 
activities, professional reading, and research) , exec u ting 
(which also embraces learning by doing through in-patient 
care, out-patient care, emergency room care, performing 
surgery, and clerical work) , and teaching . The resulting 
categorical time commitments are summarized in table 11-10. 
While there is a tendency toward decrease in formal learning 
activities and increase in execution and teaching over the 
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four years, which may suggest some sequence and graded 
responsibility, none of these differences is statistic- 
ally significant and certainly do not appear, great enough 
to have educational significance. Perhaps most surprising 
is the observation that research, which many would feel 
represents a culminating experience that is more likely to 
be fruitful if based upon a steadily widening familiarity 
with significant problems, appears to occupy far more 
time in the first residency year than in the final year. 
However, since some programs completely devote the first 
year of residency to resident research, the average amount 
of research done by first-year residents may be inflated 
by inclusion of these programs. 



* 
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Operative P r ocedures 



On the average, a resident spends 14.9 hours per week, 
775 hours per year, or approximately 3,000 hours during his 
residency in the operating room either observing, assisting, 
or performing surgery. We now turn our attention to when 
residents learned, from whom they learned, and how confident 
they arc about what they learned in this time. 

The residents are from quite varied training programs 
which have their own characteristics in terms of patient 
population, selection of cases for resident teaching 
and training, and precedence with regard to delegation of 
resident responsibility and authority. (See the next sec- 
tion for a discussion of delegation of authority.) However, 
clear patterns emerged which can. describe most resident 
experiences with certain operative and nonoperative 
procedures . 



Residents responded to five questions about each proce- 
dure by checking one of the alternatives provided.* The 
questions were: 

1. When did you first learn this procedure? 

2. How many times have you performed it? 

3. How well can you perform it? 

4. Who first taught you? 

5. if self-taught, what method (s) did you use? 

A total of 53 procedures were selected to represent 
varying degrees of difficulty and did not attempt to be 
representative of procedures that all orthopaedists should 
know. 



It is interesting to note that residents at different 
levels of training respond differently when asked when they 
learned a particular procedure. The percentage of residents 
who report having learned a procedure before beginning 



*See Resident Procedures Form, Supplement 
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residency is usually highest for first-year residents, with 
a steady decline in this percentage for second-, third-, or 
fourth-year residents. This finding would support the hy- 
pothesis that what the resident accepts as evidence of 
learning changes as he progresses through his residency 
training; he may discover during residency that he really 
did not know all of the implications and possible complica- 
tions of a particular procedure as well as he thought he 
did. This decline in perceived early competence is common 
for many of the procedures surveyed. 



Previous Experience 

There were 15 procedures that over 75 percent of the 
residents stated they learned before entering orthopaedic 
residency. These procedures consist primarily of two types 
of experiences. First, those procedures commonly delegated 
to students, interns, or first-year residents (c.g., inject- 
ing a painful joint) . Although there can be a good deal of 
discussion about how well they are accomplished, the tasks 
are generally considered to bo relatively simple and have 
a wide margin of safety, and any errors which occur can be 
modified fairly easily when seen later by a more experienced 
person. The second types of experience are those procedures 
which are ordinarily emergent in nature, and, perhaps more 
important, frequently quite urgent. A resident confronted 
with a patient requiring a tracheostomy and/or in shock, 
quite simply does not have time to seek consultation. He 
must undertake the immediate treatment and management until 
a more experienced person arrives. Such emergent procedures 
may have been learned earlier in the military or in the 
emergency room. 



Orthopaedic Training 

■ There were 34 procedures that fewer than one-fourth of 

the first-year residents reported they knew how to perform 
prior to their orthouaedic residency. A large number of these 
are more involved orthopaedic procedures that are less likely 
to have been delegated to the student or intern(e.g., perform / 

(' 

\ ( 
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a wrist synovectomy, repair a lacerated digital nerve, and 
insert a medullary rod in the femur) . Others are considered 
more esoteric in contemporary orthopaedic surgery, regard- 
less of the level of experience, (perform a needle biopsy 
of a lumbar vertebrae, insert Harrington rods, perform and 
interpret an arthrogram) . Some others are procedures in 
which the risk of the situation is such that the attending 
staff feels compelled, or are required by contemporary 
society, to undertake themselves (e.g., manipulate a con- 
tracted joint, perform an innominate osteotomy, give expert 
testimony in a court of lav;) . 

No Experience 

More than 10 percent of fourth-year residents reported 
that they never learned eight of the procedures, more than 
25 percent never learned five of the procedures, and more 
than 50 percent: never learned six of the procedures. Tho 
failure of fourth-year residents to learn these procedures 
may be attributed to several of the following causes or a 
combination thereof. These procedures are not common in 
any orthopaedic service (e.g., apply a minerva jacket, apply 
a corrective cast for scoliosis, perform an innominate os- 
teotomy) . Others are simply not done in some areas of the 
country or by some orthopaedic services because of local 
precedence (e.g., insert Harrington rods, perform a lamin- 
ectomy and discectomy, perform and interpret an arthrogram) . 

A few of these are procedures frequently done by a consult- 
ing service or a consulting surgeon (e.g., repair a lacerated 
blood vessel, perform and interpret a nerve conduction test, 
perform and interpret an EMG) . Finally, there are those 
procedures which ordinarily are not delegated because they 
are medical-legal situations in which attorneys and compen- 
sation boards prefer to have already certified individuals 
with all credentials necessary to impress an arbitrator or 
a jury (e.g., give a legal deposition, or expert testimony). 

If learning any of the procedures in this section is 
critical to the complete training of an. orthopaedist, then 
some organized method of assessing resident's exposure to 
learning the procedures should be devised. Many programs 
require residents to keep a list of the operative procedures 
which they performed or assisted at. The list is reviewed 
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periodical ly with the chief or one of the attendings to 
assure complete exposure to and learning of important 
procedures . 

The list of procedures which many fourth-year resi- 
dents have not learned, has already been employed by a 
few programs to assure the resident's exposure to the 
following: giving a legal deposition, giving expert 

testimony in a court of law, and evaluating a disability 
and reporting those findings in a form acceptable to a 
compensation board. Attendings from these programs 
realized that while these three procedures were ones that 
they perform very frequently in private practice, the 
residents-in-training ii. hospital settings have no experi- 
ence with them. One program designed a rotation in medical- 
legal problems during which a resident would write and 
evaluate disability reports, hear court testimony, and help 
prepare legal depositions given by the attending staff. 

What would possibly be considered more important by 
many practicing orthopaedic surgeons is the fact that there 
are residents completing formal training who have not done 
a laminectomy and discectomy, have not performed and inter- 
preted a myelogram, did not fool confident in their ability 
to repair lacerated blood vessels, or have not performed 
or been exposed to anterior cervical fusion. This, however, 
is a reflection of contemporary practice and in the course 
of designing programs one must constantly keep abreast of 
what constitutes contemporary practice so that the gradu- 
ating resident is not only well-prepared to meet the demands 
and challenges ordinarily placed upon him early in his prac- 
tice but is also exposed to procedures that may be in the 
developmental stage when he is in training but may become 
far more commpn after he has been in practice five to ten 
years . 

Confidence without Performance 



A number of residents reported confidence in their 
ability to perform without actually having done a procedure. ~ i 
This discrepancy between not having performed the proce- 
dure and having confidence in the ability to perform it 
seems more apparent than real. One has to realize that 
the practicing orthopaedic surgeon is constantly expected 
to be able to perform procedures or certain applications 
\ 
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of those procedures that he has never performed before. 
This implies that emphasis on basic surgical principles 
and basic knowledge is extremely important during resi- 
dency, since it would not bo reasonable or possible for 
a resident to be exposed to all the possible procedures 
he may be required to do. 

Summary 



This Study indicated a wide variety of resident 
training among the orthopaedic residency programs surveyed. 
This variety of experience with the 58 selected procedures 
exists for both residents beginning their training and 
those about to enter practice. There were orthopaedic pro- 
cedures which over 75 percent of the residents beginning 
their orthopaedic residency had previously learned. There 
were some orthopaedic procedures that over 55 percent of 
the residents about to complete their final year had not 
learned. There were also orthopaedic procedures which 
residents had never performed but had confidence in their 
ability to perform when called on to do so. 

The significance of these findings is in the example 
it provides for the method of gathering detailed infor- 
mation during residency programs about what residents arc- 
learning. it. may provide a model upon which a core curric- 
ulum of basic procedures all qualified orthopaedic practi- 
tioners must be able to perform with confidence can be 
derived and evaluated. it may provide program chiefs with 
a means of evaluating the clinical variety available to 
each of t.hoir residents. Also, it may form the means of 
discovering continuing education needs. The format of the 
instrument can provide information necessary for all the 
above purposes. 
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Respons ibili tv in Operative Procedures 

Individual operative procedures were examined in the 
previous section. The detailed steps in decision-making 
about the performance of general operative procedures is 
the subject of this section. 

Residents were first asked to define in their own words 
the two most common typos of patients in the hospital of 
their present rotation. They mentioned three categories 
most often. They are, in the words of the resident, pri- 
vate, clinic, and third-party-payment patients. 

Although residents wore directed to respond to the 
questions personally, it appears that they responded as 
residents in general, since responses of residents at all 
levels of training g were essentially the same. 



Preoperative management of patients is most often 
controlled by residents with all types of patients. Resi- 
dents have this responsibility most often for clinic pa- 
tients (03>') and least often for private patients (40vo) . 

Arranging for proper equipment in the OR is usually 
handled by the resident in 70 percent of the cases, by the 
OR staff in 20 percent of the cases, and by the attending 
in a little over 10 percent of the cases. 

The attending' s role at the operation is quite differ- 
ent depending upon the type of patient (see table 11) . The 
attending performs about half of the operations for the 
clinic and third-party-payment patients, observes one- 
quarter of them, and is not present in the OR but available 
in the immediate area for the remaining one-quarter. 

The entire operation is usually done by the attending, 
or by the resident with stop-by-step guidance from the at- 
tending, for 62 percent of the private patients and 16 per- 
cent of the clinic patients. With comments, assistance, or 
observation by the attending, the resident performs the 
operation for 27 percent of the private patients and twice 
that for the clinic and third-party-payment patients. 

Ancillary procedures as a part of a larger procedure, 
such as obtaining grafts, are most commonly done by the 
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rcsident (HO/). The resident is in complete charge while 
the a Lionel i ng may not be present, in about half of the 
canon lor clinic and third-pnrty-payment patients, and in 
one-quarter of the cases for private patients the attending 
may not bo physically present in the OR. 

Closure of l ho wound in done by the resident in 90 per- 
cent of all cases arxl the resident is in complete charge in 
over half of all cases with the attending making comments in 
loss than 20 percent of all cases and in the remaining 20 
percent flic attending gives stop-by-stop guidance. 

According to the residents who responded to this survey, 
in over three-quarters of all cases the following are dis- 
cussed with the attend, ings ; 

1. nature of actual or potential complications 

2. management of actual or potential complications 

3. rehabilitation of the patient 

4. prognosis of the patient 

Between 05 and 75 percent of Die time, the following 
are discussed: 



1. steps in the basic operative procedure employed 

2. pathology of the case 

3. degree of success of the case 

When not specifically discussed with the attending, the 
residents either had these matters enumerated by the attend- 
ing or they discussed them with other residents. In about 
10 percent of the cases those matters were not discussed. 

The least discussed item among those listed was the 
basic principles of instruments and equipment employed. In 
60 percent of the cases it was discussed by attending? or 
resident groups and in 30 percent it was not discussed at 
all. 
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The Attending Staff 



Composition and Functions 

The attending staff is likely to number at least five 
and not more than 20 attendings, the average being twelve. 
Typically, at least 10 of the 12 attendings derive all in- 
come from private practice and devote no more than 25 per- 
cent of their time to resident education. Fewer than half 
of the programs report one or more full-time attending 
staff members whose total incomes are derived from salary. 
Typically, such a staff member devotes not more than one- 
third of his time to formal teaching activities, the bal- 
ance being allocated to clinical, administrative, and 
research duties. He differs from his part-time colleagues 
in that he is more likely to function in all four areas, 
whereas at least a quarter of the part-time staff have 
neither administrative nor research responsibilities. 

For both the full-time and part-time attending staff, 
the activites related to resident education most commonly 
include ward rounds, and clinic or operating room super- 
vision. Almost as many attendings spend at least some 
time in individual conferences with residents, and approxi- 
mately two-thirds report at least some participation in 
grand rounds and special lectures. 

Tables 12 and 13 indicate that attending staff, like 
teachers in other areas of education, support contemporary 
theories of administration, planning and teaching, based 
on educational psychology principles. 



Values and Attitudes 



With respect to the goals of residency training, most 
attending staff feel that residents should be evaluated on 
attitudes as well as on achievement and that attitudes can 
be modified during residency. They are divided on the ques- 
tion of the desirability of resident research, and most 
disagree (some strongly) that the development of surgical 
skill is "the most important function of the residency." 



- 63 - 



Whi.1e attend ings tend to agree that staff should use a 
resident's previous experience in deciding what he should 
do on his new rotation, over half disagree (some strongly) 
that the goals of the residency should be determined by 
the resident's needs and inte - ests, and one-fourth disagree 
that individual differences should be accommodated in plan- 
ning the overall program of each resident. While they also 
tend to agree that "mature residents learn more on their 
own initiative than under tight supervision, " they are al- 
most divided on the question of whether "residents should 
have more supervision than they get," whether they will 
"cover important material without some prodding " and 
whether, if given a free choice of activities, "residents 
generally select what is best for them." Finally, substan- 
tial numbers disagree (some strongly) that 'residents should 
be given more freedom in designing their own programs, " and 
about three-quarters agree (some strongly) that "discipline 
is necessary for maintaining resident productivity." Al- 
though most of tnem express the view that "effective teach- 
ing is enhanced by getting to knew the residents well," at 
least half of the attending staff feel that they are most 
effective when they maintain a "proper professional dis- 
tance." between themselves and residents; at least one-fourth 
believe that an attending staff member who becomes involved 
in the personal problems of residents loses his effective- 
ness as a teacher. 

In summary, the results show that attendings are quite 
different in their perceptions about methods of learning 
and strategies for organizing and administering an ortho- 
paedic residency program. The disparity is typical of 
educators in general and indicates a need for discussion 
among the attendings in each program about the principals 
of learning and instruction to be used in their program. 

The staff development procedures discussed elsewhere in 
this report provides a media for such discussion in an 
organized series of topic areas. 
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TABLE 11-12 



More than 85 Percent of the Attendings Responding 
to the Attending Attitude Survey Agreed wi '-.h the following 
;statments : 

1. The most effective teacher has a personal interest 
in the progress of each resident. 

2. Skillfull teaching can increase a resident's interest 
in orthopaedics. 

3. The residents' service responsibilities are useful 
for learning orthopaedics. 

4. Residents learn research procedure best by doing 
research . 

5. Attendings should make allowance for individual 
difference when teaching residents 

6. Attendings who respect residents encourage partici- 
pation in planning patient management 

7. Good teaching cases relate isolated problems with 
overall goals of orthopaedic management. 

8. The contrasting skills and interests of residents 
makes team work a profitable experience. 

9. Attendings should have an overall plan of program 
objectives to use in their work with residents. 

10. Resident morale is maintained when open communica- 
tion is possible. 

11. Demonstrated interest in a resident's progress 
increases his productivity. 

12. Follow-up is as important a learning experience for 
residents as is acute management. 

13. Teaching residents is a valuable activity. 
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TABLE 11-12 (Coat'd) 



14. Residents should be evaluated on attitudes as well as 
on achievement. 

15. Attend ings should be prepared to demonstrate the rela- 
tionship between basic information and clinical problems. 

16. Residents learn best in an atmosphere of mutual respect. 

17. Discipline is necessary for maintaining resident 
productivity . 

18. A good residency program integrates the basic sciences 
with clinical problems. 

19. Attendings should participate in the teaching program 

if they wish to have residents working on their service. 

20. Patient management experience aids residents in the 
development of suitable attitudes toward patients. 

21. Residents learn most efficiently by being required to 
assume responsibility for patient care. 
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_ TABLE 11-13 

f, 



More than 70 Percent of the Attendings Responding 
to the Attending Attitude Survey Disagreed with the follow 
ing .'statements: 



lo Teaching on ward rounds is usually impractical. 

2. Teaching residents is best done by giving organized 
lectures . 

3. The quality of the teaching program has no effect on 

the residents' interest in orthopaedics. 

4. Attendings adequately meet their teaching responsibil 
ities just by providing patients. 

5. The major function of the resident is to assist in 
getting the work done . 

6. All an attending needs, to be a good teacher, is an 
extensive knowledge of orthopaedics. 

7. The residents' service responsibilities interfere 
with their learning. 

8. Developing surgical skill is the most important 
function of a residency program. 
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Ill - ANALYTIC STUDIES 



D ata Definition 

There are two examinations mentioned often in the 
following sections of the report. To prevent duplica- 
tion they will be explained once here. 

The Orthopaedic In-Training Examination (OITE) is ad- 
ministered yearly to residents in orthopaedic residency 
training programs. The examination consists of multiple 
choice questions (M/C) and written simulations of 
clinical encounters with patients, also called by two 
other descriptive terms (patient management problems 
(PMP) and erasure examination) . Erasure examination is 
a term used commonly by the resident examinees and is 
descriptive of the format of the questions. Information 
gathering and decision making involved in solving the 
problems are done by erasing areas on the examination 
booklet, thus obtaining information needed to proceed 
through the examination. The examination results are 



reported as follows: 

Multiple choice questions 

total score 

general orthopaedics ) 

adult orthopaedics ) - A 

children's orthopaedics ) 

trauma ) 

anatomy ) 

physiology-biochemistry ) 
pachology ) - B 

biomechanics ) 

hand ) 

rehabilitation ) 

recall ) 

interpretation ) - C 

problem-solving ) 



Patient management problems 
total score 
diagnosis 

treatment 

recall ) 

interpretation . ) - D 

problem-solving ) 
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In A, each multiple choice question is categorized 
into one of the subject matter areas listed. in B , M/C 
questions are categorized on other content and a ques- 
tion may appear in more than one category. The defini- 
tion of the scores in C and D is as follows: 

recall. - items which may be answered on the basis 
of remembered factual information. 

interpretation - items which may be answered by 

proper interpretation of verbal or visually 
-presented find ings . 

problem-solving - items requiring clinical judgment 
or employment of problem-solving strategies 
in addition to factual information and 
interpretive shills to answer. 

The two PMP scores are defined as follows: 

diagnosis - the culmination of history, physical exam- 
ination, laboratory, x-ray diagnosis, and 
consulatati on in which the examinee is invited 
to select as many as he would like in the 
diagnosis of patient disorders 

treatment - the culmulative decisions implementing 

treatment of patients * 

The above definitions are taken from reports of the 
American Academy of Orthopaedic Surgery. 

The second examination mentioned in this research is 
the Orthopaedic Certification Examination (OCE) . This 
examination consists of the same types of questions and 
scores as the OITE in addition to a number of cral exam- 
inations which are not discussed in this research. 

Prior to the administration of the OITE. the chiefs 
of residency training programs were requested to provide 
information about resident's achievement in selected 
areas. This Resident Evaluation Form consisted of 12 scales. 

For each scale, the chiefs were requested to compare each 
resident to the average of all residents at that level of 
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training that he has known. The scales range from 1 to 12 
with 6.0 called average. (See Supplement for sample). 



In ferpreta Mon of C or_r g l_u t.i9.ll : i: 

A correlation is an index of association between 
two variables or measures. When evaluating "the worth" 
of any correlation coefficient, there are two matters to 
consider. The first is the significance level of the 
statistic. In order to put any faith in the predictive 
usefulness of a correlation coefficient it must be large 
enough to be statistically significant. A statistically 
significance correlation is one in which there is a high 
probability that tire observed association is not due to 
chance. State. rents such as p<.05 and p < .01 indicate 

that the probability of the correlation coefficient being 
due to a chance association between tire two variables is 
less than 5 in 100 and 1 in 100 respectively. 

The second matter to be considered is the educational 
significance or useful.lness of the calculated correlation 
coefficient. Educational significance is a matter of how 
well the coefficient can serve as a predictor of performance. 
A coefficient less than .2 indicates only a slight rela- 
tionship between the two variables and is of little use as a 
predictor. In the research under consideration here are 
studies of characteristics of residency programs and 
residents, related to performance scores on the OITE. In 
many cases statistically significant correlations below .2 
were found. Such findings indicate a slight relationship 
between the characteristic and performance but certainly 
do riot indicate that these characteristics taken individually 
are useful for prediction of performance. Finally, it 
should be noLod that although variables that have a cam e- 
and-effect relationship to each other certainly will be 
expected to be correlated, the fact that two variables are 
correlated can never be used as evidence that a cause-and- 
effect relationship exists, correlation only indicates 

an association between two variables. 

• • 
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S ampl e vs Population 

In Phase II of the Orthopaedic Training Study, two 
samples of residency training programs wore selected from 
all U.S. programs. The first consisted of 200 residents 
from 16 residency training programs from which extensive 
data collection and site-visits to observe the program in 
action were to be made. This group of programs was titled 
the intensive study sample. The .second consisted 
of 504 residents from 35 programs and was titled the 
survey study sample since the residents from this group of 
programs were only to complete several questionnaires. 
Together these samples are called the Study sample compri- 
sing 784 residents front 51 residency training programs. 

(See supplement) . 

In Older to determine if achievement of the Study 
sample could be said to be representative of the population 
of all U.S. programs from which this sample was drawn, a 
study was conducted which compared the yearly orthopaedic 
In-Training Examination (OITE) scores of the Study sample 
with those of the population. 

Statistically, this procedure entailed constructing a 
confidence interval about the population mean using the 
standard error of measurement of the mean, a function of 
the standard deviation of the population, and the sample size. 

As Table 1 indicates, the study sample is significantly 
different, from the population for nine of the 11 scores of 
the OITE. The two non-significant scores were within .2 of 
being significant. Therefore, the study sample cannot be 
considered a random sample form the population but must 
rather be considered to be comprised of residents above 
average in the orthopaedic knowledge and abilities measured 
by the OITE. Since the OITE, along with ratings of resi- 
dents by their chiefs of training were to be used as a 
criterion of achi evement success in the residency training 
program, this result has great importance. care must be 
taken when generalizing test score characteristics of 

individuals in the Study sample to the population. However, 
the standard deviations of the Study sample and the population 
were nearly equal, and comparisons of characteristics of 
individual residents variation from other residents at any 
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Ye a rT v Cons i «: to ncy o f Chip f ' Hatjn^ Fnnn 

This study was conducted to compare vnr i ous ratings 
qivcn residents each year by their chief.*; of training in 
conjunction wi i.h the Or thopned i c In-Training Examination 
program. The hypothesis was that the correlation:; between 
yearly rati ire; given res i don l a would decrease as the? time 
between the ratings increases, (i.o. all resident;: would 
not .increase in their abilities equally as they progressed 
through the residency). 

As Table 3 indicates, the mean rating for all factors 
increased iron' year one *o year four by about one point, 
per y’o.ir, from 6.0 to 8.0, on a 12 point scale. The chiefs 
were instructed to compare each resident to all. residents at 
that lev-*! of training that he has known. Assuming that, 
the chii.-'.s followed the i ns l rue t j on a exactly and that the 
present »•<»;: dents wore about the name a s previou • rvuid: sts 
in the achievements rated, the average ratings at each year 
in training should be the average rating of 6.0 on the 12 
point seals. It appears that the present first-year resi- 
dent js above average (6.0 of a possible 12) considering 
all the first-year residents that the chief lias known? but 
his S( aond-yeor residents are about eight percent better 
than all second-year residents lie has known vi-o. 7.0 of 12 
or 50 percent, 8 percent bettor than average), the third- 
year residents are about 17 percent better and his fourth- 
year residents are rated as about 25 percent bettor than 
all the fourth-year residents he lias known. Another pos- 
sible* explanation is that the ciiiefs are rating each 
resident as compared to first, year residents. In this case, 
first-year residents are average at.d residents improve about 
8 percent per * 

When we turn our attention to the hypothesis under 
question, we find that for three of Lho five factors the 
hypothesis is supported, (see Table 2). The exceptions were 
the factors of overall competence as an orthopaedic surgeon 
and the knowledge of basic science as related to ortho- 
paedic:; which are significantly correlated in both compari- 
son year pairings. The significant correlation of those 
two factors in rating the same residents over time indicat.es 
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tli.il, as rated by their chiefs, residents gain similar 
amounts in their basic science knowledge and overall 
competence. For the chiefs' ratings of knowledge of 
clinical orthopaedics, ability to use information to 
solve problems, and judgment .in deciding appropriate 
v.are and l real a.cnt , although the average ratling increased 
by about oik. point from year to year, the correlations 
between the ratings decreased as the time between the rat- 
ings increased. Also, for these’ three factors, the first- 
second year correlations are statistically significant, 
while the f.irst-lhiid year correlations arc not statis- 
tically significant. It is assumed that the same chiefs 
trade the ratings for each resident at each year. 

The fact, that correlations between rating of those 
factors <i... i . used over time indicates that, as perceived 
b” t.ho chic, ‘.'s , all residents do not increase in their ach i 
merit by l ho sumo amount . If all residents were to increns 
the r uch i evcm.en t the same amount, each year, the correla- 
te m bet n yearly ratings would he very high, it appears, 
thorcfo'c, that the chiefs of training are aware of change 
of different degrees in the individual residents that they 
rate and that they do not merely consider that a year of 
experience has the same effect on all of his residents. 
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Correlat ion of Chief's R at in g s OTTE S cores 



• This study was conducted to determine the relation- 

ship between selected chief's ratings and scores on the 
OITE. It is generally thouaht that correlations between 
subjective measures, like the chief's ratings, and objective 
test scores will be low because of the expected unrelia- 
bility of subjective ratings (i. e., differences in the criter- 
ion employed by various raters), especially wich the ratings 
made by 51 chiefs. If scores on the OITE are accurate 
measures of certain achievements of residents and if inter- 
rater reliability is high, correlations with the chief's 
rating of similar achievements would be expected.* It was 
hypothesized that there would be significant correlations 
between the following rating factors and examination scores: 



Scores 

Multiple choice 
total score 

PMP diagnosis 

PMP treatment 

PMP total score 

Multiple choice 
total score 

The results were analyzed for residents at each level 
of training. There were 152 first-year residents, 393 
second-year residents, 441 third-year residents, and 382 
fourth-year residents in the study. 

The results indicate that the total multiple choice 
score was significantly correlated with all of the selected 
chief's ratings of residents at each of the four levels of 
training. These correlations ranged from .42 to .20. The 



Rating Factors 

Knowledge of orthopaedies, 

clinical ani basic science 

Problem solving ability 

Judgment in deciding on treatment 

Overall competence 

Overall competence 



See table 4 (III), p. 81. 
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patient management problem (PKP) to'*al score was 
significantly correlated v/ith all selected chief's ratings 
for second and third-year residents only. These correla- 
tions wr-re lower and ranged from .20 to .13. 

Of the particular scores expected to correlated highly 
with the chief's ratings of similar achievements/ five of 
the six were significantly correlated (see Tabic 4). As 
expected, significant correlations wore found at all four 
levels of training between the knowledge of clinical ortho- 
paedics, knowledge of basic science as related to ortho- 
paedics, the overall competence as an orthopaedic surgeon 
ratings and the total multiple choice score. For three of 
the four years; in training significant: correlations were 
found between the judgment in deciding appropriate care 
and treatment rating and the 'reatmeht PMP score. Also, 
for three of the four years in training significant cor- 
relations wore found between the overall competence rating 
and the PMP total score. 

In general, the OITE scores which summarize the resi- 
dents 1 performance on each of the two sections of the exam, 
multiple choice tot al score and PMP tocal score were the 
best predictors of all the chiefs ratings. Knowledge of 
basic science as related to orthopaedics, knowledge of 
clinical orthopaedics, and the overall competence ratings, 
in that order, are most associated with the multiple choice 
scores. Since the multiple choice questions on the OITE 
are composed of orthopaedic basic ulence and clinical 

application questions, many of w' . call for recall of 

information, it is logical that the chief's ratings of 
knowledge in these areas were most highly correlated with 
the multiple choice score. Since many cf the multiple 
choice questions, besides requiring knowledge of facts, 

•were designed to tap areas involving the interpretation 
of, judgment v/ith, and application of facts, it was 
expected that the total multiple choice score would be signi- 
ficantly correlated v/ith the chiefs rating of overall com- 
petence as an orthopaedic surgeon. 
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Knowledge of clinical orthopaedics and ove.ra.ll com- 
petence as an orthopaedic surgeon, in that order, were 
most associated with the PMP total score. Since the PMP 
■was composed of similatcd clinical encounters, it was not 
surprising that the chief's rating of Knowledge of clinical 
orthopaedics had the highest correlations with the PMP total 
score. Similarly, there was the expected significant cor- 
relations between the PMP tree tmont score and the chief's 
rating if judgment in deciding appropriate care and treai - 
meat of patients. Clinical judgment involves the aJjil.it/ 
to decide upon and carry out a treatment plan and it is 
this ability that the PMP treatment score purports to 
measure. This evidence indicates that it deals with si ,11 ar 
concepts. The expected relationship between the rating 
of the residents ability to use .information to solve problem 
and the PMP diagnostic’ score was not apparent. This cor- 
relation was very low for three of the four levels of train- 
ing. This apparent discrepancy can possibly be explained 
by the manner in which the diagnosis section of the PMP on 
the OITE is scored. It is not scored on the basis of 
merely obtaining the correct diagnosis but rather on the 
Jiasis of thoroughness of t'. ? inquiry. This distinction is 
evidenced by the fact that practioners score lower than 
residents on the diagnosis section only. Therefore, the 
diagnosis PMP score is perhaps more an evaluation of 
thoroughness of diagnosis and should not be expected to 
correlate highly with the chiefs rating of ability to use 
information- to solve problems. 

A comparison of the scores and ratings of similar 
achievements indicated the expected significant relation- 
ship. These results provide evidence for two conclu..ions. 
First, there is some evidence of chiefs' consistency in 
rating resident's achievement. If there were great integ- 
rator variety in the criterion used by the 51 chiefs in 
mahing the ratings, the ratings would not correlate with 
any other measures of similar achievement. Second, there is 
evidence that the OITE scores have some validity (i.e., they 
measure what they are intended to measure). Consistent 
with expectations, correlations were iound between ratings 
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and scores of similar achievements. This finding 
provides partial evidence of the concurrent validity (or 
.correlation v/ith other independent measures of the same 
ability) of these scores. In general, it can at least be 
said that chief’s ratings and 01Tb scores are both evalua- 
tions of similar kinds of achievement. 



4 



I 





* 



• i 



/A, 




87 



- 81 - 



t 



H 

H 

H 

W 

h-1 

§ 



w 

H 

H 

O 

r a 

c 

rd 

w 

Cn 

G CP 

g 

•H 



•H 

4J 

rd 



W 

4H 

0 

•H 

rC 

a 



■H 

rd 

P 

Eh 



P 

rd 

QJ 

!* 



g 

Q) 

QJ 
2 XI 
P C 

cj rd 
X QJ 



Cn 

c 

•H 

c 

•H 

rd 

P 

Th 

G 

•H 

P 

rd 

QJ 



co 



eg 



•X 

♦ 

♦ 

CO 

CM 



•X 

* 

♦ 

cp 

ro 



•X 

* 

* 

VO 

ro 



* 

* 

* 

VO 

<N 



•X 

•X 

•X 

in 

ro 



* 

* 

* 

o 



* 

* 

* 

eg 



•X 

•X 

•X 

O 

ro 



eg 

o 



* 

* 

in 



ro 

o 



+ 

* 

* 

in 



* 

* 



ro 

O 



OO 

o 



•X 

* 

CO 

H 



•»* 

* 



* 

* 

•X 

VO 



* 



* 

* 

* 

CN 

rg 



* 

* 

* 

r- 

CN 



* 

•X 

•X 

VO 

(N 



•X 

•X 

ro 

<N 









w 


QJ 






Td 


> 










1 rH 


1 




w 


p 




0 


0 




1 


G 


rH 










QJ rd 


0 




c 


0 




■H 


•H 




QJ 


rd 


0 


a 


QJ 






rd -P 


rd 




0 


4-1 






0 




P 




w 


•H 


4-’ 


T5 




Qa 0 


a» 




*H 






QJ 


X 






CP 




-p 


rd 


G 




0 -P 


0 


rH 


P 


W 




rd 


0 




W 


G QJ 


0 


U) 


•H 


rd 




x: 




rd 


rd 


QJ 


QJ 


C4 






rd 


•rH P 


■p 


0 


P 






-P Qa 


4J 


-P 


•H 


P 


P 


0 


Q) 






P 0 




G 


Oa 


Cp 




P s 


p 


0 


QJ 


o 


0 


X 


rH 




QJ 


rd 0 


G 


CP 


0 


G 




0 C4 


0 


-P 


P 


0 


0 


P 


Oi 




0 


P w 


0 


rd 


p 


♦H 










P 


CO 


CO 


P 


•H 




G 




•H 


♦H 


Qa -p 




G 


G 




0 






0 


P 




QJ 


W QJ 


P 


Td 


Qa 


rd 




rd G 


rd 


G 0 


u 




P 




rH 




•H 


a o 


rd 




rd 


P 




rd 




rd P 






W 


•H 


P 




a 


■H *H 


Ei 


Td 






QJ 


w 


w 


0 


Td 




QJ 


rd 


e 




w 


Td o 


P 


G 


Cp -P 


P 


rd Cp 


rd 


Cp o 


QJ 




c-t 


CJ 








QJ X 


O 


rd 


G 


G 


0 


G 




G W 


P 






*H 


rH 




u 


rd 0 


4H 




•H 


QJ 


a 


QJ -H 


0 


•H 


0 




1 


G 


rd 




*H 


Qa 


G 


CP 


Td 


G 


w 


U -P 


CJ 


P 0 


QJ 






*H 


P 




W 


0 QJ 


*H 


G 


♦H 


-P 




G rd 


G 


rd 0 


*H 




w 


rH 


0 




rd 


X rH 




•H 


u 


rd 


Qa 


QJ P 


0 


P 'H 


QJ 




P 


CJ 


P 




X 


-P Qa 


QJ 


-P 


QJ 


QJ 


§ 


-P 


-P 


0 


CO 




0 










P -H 


W 


rd 


Td 


P 


CU 


QJ C 


0 


G X 






p 


4m 


Td 




4-1 


0 P 


P 


P 




-P 




Cli U 


p ( 


0 0 






CJ 


0 


G 




c 


rH 




QJ 


G 




-P 


G QJ 


G 


0 






rd 




rd 






0 P 


0 


in p 


•H 




C 


0 CP 


0 


CP 0 






Cm 


QJ 






QJ 


4J g 


-p 


& 0 




G 


QJ 


U P 


0 


P rH 








CP 


CP 




CP 






qj a 


-P 


rd 


G 


p 




P & 








T3 


G 


QJ 


Td 


Td rH 


£>1 rH CO 


G 




-P 


H W QJ 


H 


W *H 








QJ 


■H 


P 


QJ 


QJ rd 


-P 


£ 


QJ 


QJ 


rd 


rH P 


rH 


P 








*H 


P 


0 


rH 


-P -P 


•H 


0 


& 


P 


QJ 


rd CJ 0 


rd 


0 rH 








£ 


rd 


CJ 


§ 


rd 0 


rH 


M S 


CP 


rd 


P 


p *h a 


P 


•H p 








0 


P 


W 


0 


rH -P 


•H 






a 


-P 


QJ Td W 


0 


"0 G 








G 






G 








P 






> 


> 




















< 




h> 






O 


O 





mno 
o o o 

• • • 

WV 

&i O-i &i 

•X 

•X -X 
•X -X * 



1 



o 

ERIC 



88 



:.±A 

.y)z 



- 82 - 



C h a r a c t; o r \ I. 1 r s o T R'’S i rioncv ; gram:; and Pc si dm I' s 
Crrrcl a ted w L Lh Res i d o n t :> I.;: - ami na I- i on Score:: 

1. Ort.’oopaod ic Training Program Character i.s t i.es Related 
to Resident's Performance on the OITE. 

In order to determine which characteristics of 
orthopaedic pi o'jrair.: arc* associated with resident per- 

formance on the OITE# a correlation studv was conducted. 
Only fourth-year ran idem is from (.he Studv sample woo 
employed in l he* analysis, since effects of the program 
characteristics would tend to be more apparent in their 
performance than in that of residents at other levels 
of training. There were 428 fourth-yon r residorU-’ 
included in the study. 

There was a significant correlation between the 
multiple choice total score on the O'.TE and each of 
the- progra m characteristics listed. These correlations 
ranged from .20 to .13. Higher multiple choice scores 
were associated with residents from prog i am s with the 
folio*, ing characteristics: 

Ch i c f 

1. the chiefs major office was in or very near 
the principal institution of the program 

2. the chief had much of his total income from 
salary 

3. the chiefs of service held academic rank 
Resident Selection 



4. resident selection was d >ne primarily on the 
basis of a candidates academic standing in 
medical school, his intellectual approach to 
problems, and his recognized interest in 
research 

5. resident selection was r o t primarily done on 
the basis of technical aptitude or emotional 
maturity 
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0. applicants for residency who were vulnerable for 
mali.tary induction during the residency were 
acceptable 

7. preference was cjiven to graduates of the affiliated 
medical school ( :i f there was one) 

Objectives 

8. the primary objetb i vo of the program wan the pro- 
duction of tlie commun ! ' *» p notice oriented ortho- 
paedists 

9. the primal y object i ve was not the product. i on of 
orthopaedic subcp 'ci al Lots or academicians 

P ost d o n t s 

10. there were many residents in Ur: program 

11. residents instructed medical students 

12. resident:, were required to write a Cx ini cal o - 
laboratorv research paper 

13. after compl e L inq the procjram residents did not stay 
on in supervisory positions 

14. some, but not many, residents have failed to com- 
plete the program in the last five years 

15. residents who failed to complete the program were 
judged unacceptable because of poor clinical judgment 

16. during the first two years of residency, residents 
salary was below $7500 per year 

17. there was supervision provided the resident in 
the operating room according to the needs of the 
individual resident rather than supervision in all 
cases 

18. there was a fellowship stc ff 

Pro gram F aci 1 i ties 

19. clinical material was more often adult and pedia- 
tric, as opposed to trauma 

20. a biomechanics subspecialist was on the ortho- 
paedics staff 
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21. a neurologic disorders sxibr.peciali.sL van on the 
orthopacd i ch si. a f f 

22. there was a veteran. ' hospital associated with 

the program • 

Tito converse of each of the proa ram charact er i sLi cs 
is associated with lower multiple ch« ice total scores, for 
example: 

1. Chief's major office was not near the principal institx.- 
tion of the program 

4-!3. res i de n L sol oc t i on was no t done y. r i mo r. i ly on t' te 
basis of academic standing in medical school, etc. 
but , -ts dene primal il w on the basis of technical 
aptitude iid emotional maturity 
10. there v/r *•-. r - j y. nts in the program 

10. clinical m iter i al was more often trauma than adult 
ai d ped La Lr i c 

Characteristics not mentioned such as the number or 
attending?: who teach residents, kinds of clinics aval Ip It. \ 
the use of a follo*-»-up clinic etc. were not si gni ficantly 
related to resident's OITU scores. 




2. Resident background related to OITF, scores 

This study was conducted to determine relationships 
between the background of orthopaedic residents and 
their performance on examinations. r i'. ere were 299 
fourth -year residents included in the stxidy. Signifi- 
cant con elations ranged from .19 to .15. Residents 
with the following characteristics tended to have high 
multiple choice OITE scores: 



younger than average 

decided on ortiiopaedi.es as a specialty at an earlv 
age 



1 . 

2 . 
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3. had a high grade point average in high school, 
college, and medical schr i] 

4. had a irigh rank in class in high school and 
college, and medical school 

5. liked mathematics in both high school and college 

6. selected their orthopaedic program because of its 
ropu ation and not because of anv offer of 
financial support 

7. was satisfied witli the general reputation of the 
training program 

8. was dissatisfied wrth b^e opportunities for 
surgj al experience to date 

9. was dissatisfied witli the amornl of responsibility 
they are given in th<$ treatment of patients and 
v/ish they hod more responsibility 

1°. marital problems were not an interference with 
work in the residency program 

On the other hand, low scoring residents can be said 
to terd in the opposite directions on all of the characteris- 
tics listed above. For example, the residents who scored 
low on the multiple choice section of the OITE tended to 
be eider, had a lower grade point average and rank in class, 
were cot satisfied with the general reputation of the pro- 
gram, and marital problems interfered with thei- work in 
the residency program. 

Characteristics not m ntioned, such as size of home 
town, parental occupalic’ or amount of farri' v income, size 
of high school, college, or medical school, ^.ype of inter- 
ship, etc. and others listed on the background questionnaire 
, (see supplement) were not significantly related to multiple 
eiioi.ee scores. 



3. Resident attitudes related to OITE scores 

The significant correlations between fourth-year 
residents' attitudes and the multiple choice total score 
were few and negative. A negative correlation indicates 
that agreement witli the attitude statement is related to 
low multiple choice score and disagreement is related to 
high multiple choice scores. There were 265 fourth^-year 
residents in tills study. 
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The negative correlation of mult iple choice r cores with 
the first five state",, on t s in Table r > iridic. nlo that low 
scores were ass.ciat»*cl with resident,:* who complain nbou' 
types of patient:; on their service (items 1 and 2) and 
about, the large number of demands upon them with too 
little time to fulfill them (items 3, 4, and r >) . 

Low multiple choice scores were also associated wi t:h 
resident's satisfaction with their oppoj tun i t.v to use 
their operat ve skills (item <S> . Conversely, high scor- 
ing residents were likely t:o state that t.hov have not had 
such opportunity. Supplementary data was found in the 
background questionnaire. ..hen ask d to evaluate certain 
aspects of thei r residency, hiqh scoring residents again 
tended to be 'Unsatisfied with the opportunities thee have 
had for surgical expo r * %. nee* while low scoring resident:' 
were satisfied. 



In t e rp ve t;: h bon o f pesul s 

In the results reported, the ^ori el at. Jons ranged from 
.20 to .13, and were therefore a.’l 5 tat ‘ sticall v signifi- 
cant. Tire educational significance of the results can be 
questioned since tire correlations are low in tenm of the 
ability to predict. CITE scores bv oLscr”ing one of tlie 
characteristics reposed in a particular p'ogran. It should 
be noted that the mere presence of any characteristic does 
not guarantee that residents will have high scores, and 
vice versa. Although statis really significant these cor- 
relations are low and indicate only a slight relationship 
between the characteristics and scores. It may also be 
that the high scores are associated not with the particular 
characteristics listed but with other characteristics associa- 
ted with them. For example, a first and second-vear resident 
salary of less than $7500 was associated with high scores. 
However such salaries are also associated with public insti - 
tutions, as university programs usually are, which have been 
shown in past studies to be associated with higher scores. 

The results surely* indicate the multi-faceted nature 
of program ef fecti veness in the sense that there are many 
and varied individual characteristics of programs which 
were all. related to program effectiveness. A further study 
of these program characteristics is described later in 
this report which employs many characteristics of the 
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proqram to d< :t r . r r ■ i n c: if any cor hi p .t ion of then. pro- 
vided a good basis for predict b.n of OlTf, scores. The; 
value o' ‘hi; present stud" is that it. i iulio.il os certain 
charael < r i si i r-. of programs that ran bo included in the 
prod i o 1 i on w i udv . 
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Cha r ad eg ; s i icy, of C' ■ t « ^ » " i * ) ;.« t os fee- j'ar l' ; Board f'or t.i Miration 

Among its ob i or i V's , l’ir o i t ’ lopried i r Training 
Study was dosiqnid "to provide a model of individualized 
gradual reheat Ion in mod i r i no • n winch t ho do-i.onsf ra- 
ti on of i nd i v : d'-'tl revi pc 1 r-rn.-r, ) at ! oj Hum tar fulfill- 
ment. of rigid time ro-.ni i i omen , i ark the end of for; cl 
t r. ' n i ng . " Unde -r Imr:s of the ;dy nrcip;., resident s 
fro... the i n 1 1 i).. i vr study .• ii'plc: -it progra: ;; wore gj-s-n ! he 
opporlun i l.-/ t.o part., i r i pn It; hi tin: certification process of 
1 1 i o American hoard of Or (hop .«>d i c Surger'* when the •• and 
the i i chief of t.i lining agreed that the'- were pj-rpml t o 
tala, the ci i is, t ion. 

'J ' ■ o pu, i\i i o of this fit nd'* v/a s to do !. o im. i no i r thee 
a'lbroviat eb re', ideney ) oa ret caudielttt.es (7\RBC’s ) d i f h v from 
their fellow res id on t p , Anv d i f feronees found could be 
used to identify and select either potential ARBCs in resi- 
devy wit’- an .attenclrnL reduction in training period,. 

Such a process would '.ncrcase the supply of orthopaedic 
manpower av; ilaele in the health care delivery s/stom. 

In 1970 and 1971 a total of 23 residents from among 
the 331 residents in the 1G experimental pi'o grams exercised 
the option of taking the certification exaai early. of these 
ARBCs , nini. were fourth-year residents and 14 were third- 
year residents at the time of the exam. No special pro- 
vision for, or special designation of these men was made 
during the certification process. Examinees were only 
aware that ARBCs might be examined. All 23 attained scores 
considered by the American Board of Orthopaedic Surgery as 
satisfactory for certification. 

Data from the background survey indicate that most of 
what has been said about the tvpical orthopaedic resident 
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(see resident background section) also applies to the 
ARDC group. However, the ARBCs wore slightly younger, 
have fewer relatives who were physicians and . fewer 
relatives in the health-related fields. They came from 
larger towns than their poors, their parents were slightly 
better educated, had higher incomes and more of them were 
self-employed than the parents of other residents in the 
Study sample. While fewer of the ARBCs considered occupa- 
tions other than medicine, their choice of medicine as a 
career was made for similar reasons; desire for indepen- 
dence, interest in people, curiosity about the human body; 
but the ARBCs reported far greater interest in the profit, 
as a reason for entering medicine, than did other resi- 
dents surveyed. A larger percentage of ARBCs received 
appointments in the residency program of their first choice 
but were more anxious about their appointments than other 
residents. ARBCs perceived far more competition in college 
than their poors and slightly more competition in medical 
school and in residency, although both groups report that 
residency s about as difficult as they anticipated. Dur- 
ing the program, ARBCs differ from their poors in reporting 
less interference in training program atmosphere from rela- 
tions with at tendings ; considerably more satisfaction with 
the academic enthusiasm of their program, and with the in- 
terest. of residents with resident morale. They also re- 
pc rted slightly lower educational debts than did other resi- 
dents; but they anticipated making five to 10 thousand dol- 
lars less per year after five years of practice. This 
estimation of income was possibly a realistic expectation 
in view of the fact that many of them report an interest 
in an academic career in medicine rather than straight 
private practice. 

The OITE was conducted to provide multiple choice 
scores in many disciplines and taxonomy scores, and scores 
on patient management, problems designed to simulate the 
encounter of the physician with the patient in a problem- 
solving situation. Results indicated that the ARBCs per- 
formance on all aspects of this exam was superior to their 
peers at each year in training. 
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II: should bo noted that while ARRCs maintained tbei r 
superiority over the average resident in each of the 'ears,. 
, thei r percent: i le rank decreases as they progress through 
the residency. This indicates that the poors were catch- 
ing up slowly, since thev held more improvement to make. 

This phenomenon is also known stati s Li call v to occur 
often i.n multiple measurements of any group and is 
called regression towards the mean. 

The 1970 A It PCs have higher percent ilc ranks than the 
197.1. AUBCs . This wav be a result of the cautiousness and 
anxieties of the chiefs and residents >n 1970, the first- 
year residents have over had the opportunity to take the 
OCE earlier than usual. The success of the 1970 AUBCs mav 
have lowered the criterion, and certainly lowered the 
anxiety involved in the decision of chiefs and res 1 ' dents 
to participate in this experimental program hi 1971. 

The AUBCs continued to demonstrate their super - ' or 
performance on the 0 r 1 hopaed < c Certi.fi cation Exam (OCE). 

On the major Lest scores, total multiple choice and PMP 
total as well as all sub-test scores, the percentile rank 
of the AUBCs ranged from the 54th to the 74th with the 
majority above the 70th percentile. In general, it appeared 
that the AUBCs began their residency by demonstrating their 
superior ability on the OITE examination and this ability 
continued to be evidenced throughout the residency to and 
including the certification examination. 



Prior to the annual OITE the chief of each training 
program submits a rating scale evaluating each resident in 
his program on overall competence and performance factors. 
After a consolidation of similar ratings, the comparison of 
these yearly ratings is presented in the table which follows. 
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It appears that the; chiefs of residency training programs 
have been sensitive ^ven in the first year to the? ARBC’n 
superior performance. AKBCs an. superior as rated by chiefs 
and in scores on the OITK . 

AR PC Q n as I; ion na ir e 

The final portion of this report is based on a 
personal survey questionnaire completed by each of the AEBCs 
concerning their carc:or plans and preparation for the exam- 
ination which was distributed after the certification exam 
was given. Those data are being reported separately because 
they concern projections about future activities rather 
than present or part situations. Also, they arc descrip- 
tive rather than comparative. 

Of the 2 3 7\RBC ' s 25 percent attended review courses 
prior to the OCK; of these, only one felt the- oc-.irrse was 
not helpful. All AP.DC's reported reviewing at least five* 
years of the Bono and join t Journal and reading two or more* 
specialized texts in orthopaedies or related basic sciences. 
Only one man reported the OCE to be mare difficult than 
expected and 25 percent indicated that it was somewhat 
easier than had been anticipated. 

Despite the fact that the ARRCs were Board certified 
and not required by the Board to complete their residency, 
all but three chose to complete it. At the time of the 
survey, two of these indicated that they would enter private 
practice, and a third would utilize his remaining residency 
year in special fellowship training. The fact that most 
residents have a contractual obligation with a hospital did 
hot seem to be an interfering factor in the decision making 
process reported by the residents. The primary reason reported 
by 20 of the 23 ARBCs for remaining in the residency is 
related to a feeling that traininq was still incomplete, a 
desire for special training, an obligation to the program, 
or the desire to complete the best rotation (usually chief 
resident) . 

Practice plans of the ARBC ' s are varied, with most (10) 
anticipating entering a single specialty practice group. 





ERJC 



-94- 



Sinee ARBCs had attached greater importance to the profit 
motive as n factor in their career choice, one would expect 
that anticipated practice income would bo higher than other 
residents. Instead, their anticipated incomes are con- 
siderably less. This is, perhaps, a realistic expectation 
since 22 of the 23 ARBCs indicated n desire to do some 
teaching and four indicated a choice of full-time academics, 
which i.s probably not ns monitarily rewarding ns community 
practice . 

It would appear that the major difference between ARBCs 
and their peers is, not in their general background, but 
their performance, both on examinations and in the medical 
care system. They have demonstrated superiority throughout 
their training period. 

Such information, however, should not obscure the fact 
that 23 men were able to successfully complete the certification 
examination as much as two years earlier. This, coupled 
with the data reported, suggests three conclusions: 

First, the results support the rationale behind the 
original objective of individualized instruction, i.e., 
apparently individuals learn at varying rates and, having 
established a criteria of competence, some can be expected 
to attain that level of required competence sooner than 
others, as indicated by the fact that they all passed the OCE . 

The second conclusion is related, in that once competence 
rather than time remains the variable factor, significant 
manpower gains can be realized through early certification. 

It may also mean that the trend toward increased super- 
specialization may be accommodated without significantly 
increasing the length of time a man spends in training. 

The very talented may complete regular training early and 
move into super-specialty during the period which would other- 
wise be occupied by the basic program. 

The fact that 20 of the 23 ARBCs chose to complete 
residency training, may not be as much of an argument 
against early certification (i.e., the man is going to stay 
around until he fulfills his training time anyway) , as much 
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nf. an example of the need for planning and counseling resi- 
dents. It also points out the need for some moans by which 
potential ARBCs can be identified early in their training 
so that career plans, expectations, modifications and 
alternatives can be developed with those potential candi- 
dates. Such planning may well include discussions of the 
residents' services. 

Third, cipparontly the chiefs of residency training pro- 
grams have been sensitive to superior performance early in 
the residency, as indicated by the ratings on the resident 
evaluation form. Whan this is coupled with the fact that 
in all 23 cases the judgment of the chief and resident was 
vindicated, i.c., all 23 men who were thought to be capable 
of passing did so, considerable weight is given to the idea 
that the chief is an excellent judge of resident performance. 

The Study suggests, also, that both the OITE and the 
Resident Evaluation Form appear to be consistent predictors 
of potential success and provide effective bench marks by 
which early candidates may be identified. Obviously, a 
great deal of additional study needs to be done to determine 
what other devices for prediction arc available. 

It seems rather clear from this study that carefully 
selected individuals, obviously different from their peers 
in performance on examinations and abilities as evaluated 
by the chief of their program, can profit from an opportunity 
to complete the certification process prior to the normal 
time requirements. The Study also seems to suggest that 
the estimation of ability and prognosis for success, when 
made jointly by the chief and the resident, is highly accurate. 

Orthop aed ic In Tr aining and Certificatio n Examination 
Multiple Ch oice S cores 

The designs for both the OITE and the OCE came from 
Phase I of the Orthopaedic Training Study. Both exams employ 
a series of multiple choice questions in ten areas of 
orthopaedics , patient management problems, and ratings by 
the chief of the resident's training program. In addition, 
the OCE contains several more ratings by associates of the 



candidate and a number of .individually administered oral 
examination:!. Th‘n study examined the correlation between 
si.mi.lfir scores on each cxnninn l..:i on . Only the mult iple 
choice scores wore examined since PM I’ scoring was different 
on the two exams and could not be: directly compared. 

Although the basic design of the two exams wan similar, 
d ; f fc rupees in construction, residents preparation, and 
administration exist. The OITE is constructed by a com- 
mittee of the American Academy of Orthopaedic Surgery and 
the OCE in constructed by a committee of th'** American Board 
of Orthopaedic Surgery . The Academy constructs the 01 TE 
as a diagnostic instrument, meant to indicate residents' 
relative strong I" hr. and veal nesses . it is therefore composed 
of questions with a wide rang: of difficulty E o that one cun 
clearly d i s tinguisli between high abil i ty and low ability 
residents . The Board constructs the OCE as a certification 
instrument and it is therefore criterion oriented. Knowledge 
and abilities thought essential for any competent ortho- 
paedist. arc* included and all candidates are expected to 
obtain n certain minimum score. For the 01TE residents 
often make no advance preparation and therefore test scores 
arr reflective of their long term knowledge. On the other 
hand, many resident candidates for the OCE attend courses 
designed to prepare them for the examination. The candidates 
often review past years of the Bone and Joint J ou rn al and 
read through a text on orthopaedics. The OITE provides 
precise directions regarding administration of the exam but 
there is evidence that these directions are not followed as 
closely as they are for the OCE, since there are many 
different examination settings for the OITE. 

Considering all the differences between these examina- 
tions it was significant correlations between 
similar scores on each exam were found. The OCE scores on 
the 1971 exam were matched with the fourth year OITE scores 
for 321 residents. A correlation of .14 is significant 
at the .01 level. 
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IV - PREDICTION STUDY 



Tli in final section of the- report: is an attempt t:o 
predict resident performance on the multiple choice and 
PMP sections. of the OITE from knowledge of resident 
background characteris Li os , his attitudes and perceptions 
about h in residency program, and charnclc-rist ics of that 
program. The significance of thin analysis lie:; in the 
inve.n Ligation of Lho inforp] ;.a- between the personal char- 
acteristics and previous history of the resident and the- 
subsequent program structures and experience:; which he 
will encounter in residency. Essentially, l.n.ir> section 
attempts to examine’ the program environment for which a 
resident, is most suited, where; most suited is defined in 
terms of OITI! performance only. 

The u'.o of OITE scores as a criterion for judging 
resident performance has certain limitations. The actual 
surgical competence of a resident is virtually impossible- 
to tc-s ( by timed paper and pencil tests and is conse- 
quently not considered in this analysis. Secondly, a 
test is a somewhat artificial situation in which some 
different skills from those utilised .in the practice of 
orthopaedics arc employed. The ability to function under 
time restrictions, to recall information from memory, and 
to choose from presented alternatives are advantages in 
the OITE which do not necessarily play a role in the prac- 
tice of orthopaedics. Despite those limitations, the 
fourth year OITE scores were felt to yield the most valid 
evaluation of orthopaedic education, and were the most ob- 
jective criteria upon which to assess resident education. 



Sample 

All residents in the fourth year of residency from 
the Study sample for whom scores were reported on the mul- 
tiple choice and PMP parts of the OITE were included in 
this analysis. There was a total of 419 residents with 
the following program designations; 
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1. 373 Res i dents from University programs (89/,)^ 

2. 13 Residents. from University-af filiated 

programs (3%) 

3. 33 Res idonts from independent programs (0/) 

An As apparent from Lhc.no f iy tiros , Lhi r, analysis 
will be heavily influenced by resident... front university 
programs and this should bo kept, in mind in interpreting 
the .subsequent, results. In order to compensate for the 
university program emphasis, separate analyses will a], so 
be included which are restricted to residents from inde- 
pendent programs. 



Met hod 

Items front the various Orthopaedic Training Study 
questionnaire' wore rejected no possible sources re- 
lated to variation in performance on Liu: 0IT1J. The list 
of items used and the questionnaires from which they 
were taken is presented in Table 1. (The Supplement 
contains the questionnaire from which the items were 
taken) . If a resident had not responded to items which 
were used in this analysis, the average response for 
that item replaced the missing information. 

The statistical technique used in this analysis is 
multiple regression, which is essentially a correlation 
between a combination of variables and a criterion. Ac 
with the previous analytic studies, a correlation indi- 
cates the strength of an association between variables, 
and in the case of regression, the strength of the associ- 
ation between a com binatio n of variables and a criterion. 



^Thc definitions of these terms is j:>rcscntcd in the 
section of the Final Report which is concerned with Pro- 
gram Characteristics. The definitions appear as the first 
three footnotes of that section. 
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It should he remembered, however, that association among 
variables is not indicative of causal relationships. 

Also, in this analysis, there will be no significance 
levels presented because all reported variables and their 
combinations were significant at the .05 level. That is 
to say, that the probability that the associations ob- 
served among these variables was due to chance was less 
than five percent. 

Regression has the advantage over correlational 
analysis of permitting an assessment of variables in 
combination as predictors of a criterion. Briefly, the 
technique determines which variable is most highly related 
to the criterion, then the variable which is next most 
highly related given the influence of the previous vari- 
able, etc. In this way, unique combinations of variables 
can be abstracted which allow more precise estimation or 
prediction of an individual's performance on the criterion. 



Results 



TOTAL MULTIPLE CHOICE SCORE 



A standard score on the total multiple choice sec- 
tion of the OITE was computed for all fourth-year residents 
who had taken the examination. Th^ sample had a mean of 
50.0 and a standard deviation of 10.0. The mean was 52.1, 
with a standard deviation of 9.5 for the 419 residents 
included jn this analysis. With this measure as a cri- 
terion, seven variables were found to be optimally func- 
tional for prediction of performance on the OITE multiple 
choice section. Table 2 presents these items in descending 
order of their relative power for prediction, along with 
the cumulative prediction power of the preceding variables 
in the analysis. That is, the ability to predict increases 
as more variables are included in the analysis and the 
third column of each table presents this cumulative 
increase . 




TABLE IV - 2 



INTERACTION OF RESIDENT AND PROGRAM CHARACTERISTICS 
IN PREDICTING MULTIPLE CHOICE SCORES 



Item^ 


Positive Item 
Response 


Cumulative 

Predictive 

Power 


Admissions Criteria- 






Technical Aptitude 


Less Important 


7.26 


Type of Program 


University 


11.69 


Too many ward cases 


Disagree 


15.06 


Surgical Supervision 


When needed 


18 01 


College Grade Point 






Average 


High 


20.72 


Admissions Criteria- 






Intellectual Approach 






to problems 


More Important 


22.99 


Not enough elective time 


Disagree 


24.87 



This table shows several trends. First of all, with 
the exception of college grade point average, all : ems which 
are significant in the prediction of total multiple choice 
score are program variables or attitudinal sets which are 
developed in response to a program environment. Secondly, 
residents in programs which seem to emphasize technical apti- 
tude tend to perform less well on multiple choice sections, 
while residents in programs stressing intellectual endeavors 
perform better on the OITE. This may be due to the relative 
importance of orthopaedir 1 surgical competence in the former 
programs in compari-'^ n to aho importance or ''took" learning, 
whicn is Lhe th*- uC a of the OITE. Finally, in programs where 
residents perceive strong service demands, there tends to 
be a lower score on the multiple choice part of the OITE. 

This finding may be attributable to an actual lack of time 
to devote to book learning or it may be that these attitu- 
dinal items are measuring underlying dissatisfaction with 
the programs or with orthopaedics in general. 
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All items are included in the Supplement. 
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Having now investigated the interaction between charac- 
teristics of the resident and program influences, the rela- 
tionship of program characteristics, when the background of 
the resident was controlled for in the analysis, was examined. 
That is, abstracting out the influence of any personal char- 
acteristics of the resident, such as, college grade point 
average, the most salient influences of the programs on OITE 
performance were investigated. Table 3 presents the results 
of this analysis. 



TABLE IV - 3 



PROGRAM CHARACTERISTICS RELATED TO MULTIPLE 
CHOICE SCORE PREDICTION 



Item 


Positive Item 
Response 


Cumulative 

Predictive 

Power 


Admissions Criteria- 


Technical Aptitude 


Less important 


7.26 


Type of Program 


University 


11.69 


Too many ward cases 


Disagree 


15.06 


Surgical Supervision 


When needed 


18.01 


Admissions Criteria- 
Intellectual Approach to 


problems 


More Important 


20.13 


Not enough elective time 


Disagree 


21.92 


Use of Operative Skills 


Disagree 


23.21 



It is clear from this analysis that, despite the unique 
personal characteristics of the resident, program structures 
and resident attitudes formed in response to those struc- 
tures are predictive of OITE performance. This result, may, 
however, be slightly arti factual, due to the heavy weighting 
of university program residents in this analysis. That is 
to say, the policy of surgical supervision as needed and the 
de-emphasis on technical aptitude as an admissions criterion 
are characteristics of university programs in general and 
these character isuics may be expected to give separate re- 
sults within this sample. Further substantiation for thy 
argument is the relative importance of university progr'/ 
as predictive of resident OITE performance, as indicat 
this table. 
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In order to dissect further this apparent strong influ- 
ence of program characteristics on.OITl 1 ! performance and to 
investigate the possible artifactual nature 6f these rela- 
tionships, an analysis was performed separately for residents 
in university and independent programs. ^ The 373 residents 
in university programs had an average CITE total multiple 
choice score of 53, and a standard deviation of 9. Table 4 
presents the significant influences on OITE performance for 
university residents. 



TABLE IV - 4 

CHARACTERISTICS OF RESIDENTS AND PROGRAMS 
RELATED TO OITE MULTIPLE CHOICE SCORES 



(University Programs) 



Item 


Positive Item 

Response 


Cumulative 

Predictive 

Power 


Admissions Criteria- 


Technical Aptitude 


Less Important 


6.62 


College Grade Point 


Average 


High 


11.07 


Too many ward cases 


Disagree 


14.88 


Surgical Supervision 


When Needed 


18.53 


Impossible to meet 


demands 


Disagree 


20.40 



It is interesting to note that, when the type of pro- 
gram is held constant, the influence of a background char- 
acteristic of the resident, his college grade point average, 
is more strongly predictive of subsequent performance than 
are program characteristics which are influential when the 
sample was more heterogeneous. There are at least two pos- 
sible interpretations of these results. The environments 
of university programs may be sufficiently homogeneous that 
any differences between them are unimportant in predicting 



^University-affiliated program residents were excluded 
from a separate analysis because there were not a sufficient 
number of them. 
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subsequent performance. That is to say, it may be that the 
only relevant program characteristic is the overall type of 
program rather than the variations within a specific type 
of program. 

On the other hand, it may be that university programs 
are able to be more selective in their recruiting practices 
than are orthopaedic training programs in general due to the 
greater demand for positions in university programs. This 
would allow university programs to choose residents who would 
be more suited for success in their programs. As a conse- 
quence, the prior achievement of the residents would become 
the most relevant predictor of future performance, with the 
unique configuration of program structures among these pro- 
grams being less relevant. If admissions requirements are 
more competitive, it would probably follow that the residents 
chosen would be more well-suited for the demands of the pro- 
grams and consequently, prior achievement would be most impor 
tant in predicting future achievement. 

The 33 residents from independent programs had an aver- 
age OITE multiple choice score of 45 and a standard devia- 
tion of 10. The most significant characteristics in predict- 
ing their OITE scores are displayed on Table 5. 



TABLE IV - 5 



CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATED 
TO OITE MULTIPLE CHOICE SCORES 
(Independent Programs) 



Item 



Positive Item 
Response 



Cumulative 

Predictive 

Power 



Papers at annual meetings 
Perceived residency com- 
petition 

Full-time attending 
staff in clinical work 
Time spent in performing 
surgery 

Admissions Criteria- 
Technical Aptitude 



Yes 



38.73 



Less 



57.55 



Greater 



64.90 



Less 



69.11 



More Important 



72.95 
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Before discussing possible explanations of the results 
from this table, it is important to note that the increase 
in variance accounted for in this analysis is partially a 
function of the smaller number of subjects in independent 
programs. That is, the statistical manipulation enables 
better prediction when sample size is small, rather than a 
necessarily better understanding of the multi-faceted nature 
of performance within independent programs. 

Residents who are in independent programs which are 
less competitive, more clinical and research-oriented, em- 
phasize technical aptitude, and de-emphasize the performance 
of surgery receive higher OITE multiple choice scores than 
residents in other types of independent programs. The back- 
ground characteristics of the resident do not seem to influ- 
ence his subsequent performance. Rather, the environment of 
the program and his perception of that environment appear 
to be the strongest predictors of OITE performance for resi- 
dents in independent programs. 

There are three additional informative findings from 
these separate analyses which should be noted. First, resi- 
dents, on the average, in university programs perform at a 
statistically higher level on the OITE than do residents in 
independent programs. Secondly, the only program influence 
common to the prediction of OITE scores between the two 
types of programs is the importance placed on technical ap- 
titude in the selection process. However, with respect to 
this influence, the import of this selection criterion is 
reversed for the two program types. Finally, the lack of 
perceived competition in independent programs is positively re- 
lated to subsequent performance. Conversely, an argument could 
be made that in university programs, the more competitive 
environment, in terms of selection, results in a greater 
emphasis on prior achievement as predictive of residency 
success, as measured by the OITE. 



TOTAL PATIENT MANAGEMENT PROBLEMS SCORES 

As with the multiple choice scores, a standard score 
was computed for all fourth-year residents. The group had 
a mean of 50.0 with a standard deviation of 10.0, while 
the 419 residents in the analysis had a mean PMP total 
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score of 51.0, with a standard deviation of 9.0. Table 6 
displays the influences most strongly related to performance 
on the PMP section of the OITE: 



TABLE IV - 6 



INTERACTION OF RESIDENT AND PROGRAM CHARACTERISTICS 
IN PREDICTING PMP SCORES 



Item 


Positive Item 
Response 


Cumulative 

Predictive 

Power 


Outpatient Trauma Respon. 
Admissions Criteria- 


Earlier 


1.69 


Technical Aptitude 


Less Important 


4.14 


Surgical Supervision 


When needed 


6.29 


Affiliated Program 


No 


7.84 


School Learning 


More time 


9.02 



It is clear from this table that personal character- 
istics of the residents do not influence the prediction of 
PMP total score. Rather, the extent of independence and 
responsibility available to a resident are the most signif- 
icant estimators of problem solving ability as measured by 
the PMP's. It is also significant that two of the items 
which were predictive of multiple choice scores were also 
predictive of PMP scores, technical aptitude and surgicnil 
supervision. This may not be particularly surprising as 
the two tests scores are positively correlated (.32), in- 
dicating that certain program structures may be mutually 
beneficial in influencing both "book learning" and problem 
solving ability. That is, as there is a positive associa- 
tion between PMP score and multiple choice score, it may be 
that some of the same program characteristics that are asso- 
ciated with high performance on the PMP are also associated 
with high performance on the multiple choice section. 

As none of the items related to personal characteris- 
tics of the residents aided the prediction of PMP perfor- 
mance, it was unnecessary to control for these influences 
in a subsequent analysis. Instead, the analyses were per- 
formed separately for university and independent programs 
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to determine if the ability to predict PMP score would be 
improved by examining the two types of programs separately. 
The results of the analysis for the residents in university 
programs are presented on Table 7. 



TABLE. IV - 7 

CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATED 



TO OITE PMP SCORES 
(University Programs) 



Item 


Positive Item 
Response 


Cumulative 
Pred ictive 
Power 


Surgical Supervision 


When needed 


3.18 


Admissions Criteria- 


Technical Aptitude 


Less Important 


5.61 


Complex Procedures 


Earl ier 


8.01 


High School Grade Point 


Average 


Higher 


9.23 


Teach Allied Health 


Personnel 


No 


10.31 



Again, it is interesting to note that program influ- 
ences appear to be more significant in predicting performance 
than personal characteristics of the residents. This is 
true, even when the program type is held constant. On the 
other hand, when the same analysis was performed with total 
multiple choice score as the criterion, the influence of 
college grade point average was the second most relevant 
predictor of performance (see Table 4) . in this analyses, 
high school grade point average was the fourth most relevant 
predictor of PMP performance. This is interesting both be- 
cause its predictive power is considerably less than was col- 
lege grade point average for the corresponding multiple choice 
analyses and because this is the first time that the import- 
ance of high school grades have been observed in any of the 
analyses. It is also clear that the ability to predict PMP 
performance is considerably weaker than the ability to pre- 
dict multiple choice performance because the total cumulative 
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prediction power with PMP as a criterion is 10.31 as compared 
to 24.87 with multiple choice score as a criterion. 

Table 8 displays the results of the analysis for resi- 
dents in independent programs. Again, a word of caution is 
necessary in interpreting these results, due to the small 
number of residents included in the analysis. 



TABLE IV - 8 



CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATED 
TO OITE PMP SCORES 
(Independent Programs) 



Item 




Positive Item 
Response 


Cumulative 

Predictive 

Power 


Specific Objectives 


known 


Agree 


23.62 


Impossible to meet 


demands 


Agree 


38.79 


Resident training super- 
vision 


Close 


46.82 


Board requirements 
rigid 


are too 


Disagree 


54.42 



With the exception of the supervision of resident 
training, the attitudinal dispositions of the residents in 
independent programs seem to be most influential in pre- 
dicting of PMP performance. If program objectives are known 
and Board requirements are deemed sufficiently flexible, 
residents tend to perform well on the PMP section of the OITE. 

The findings with respect to supervision and work de- 
mands are suggestive in that within an independent program 
environment these findings may be a reflection of the cali- 
ber of the program. If residents perceive impossible demands 
and if the program reports close supervision, then the educa- 
tional environment for these residents is atypical of inde- 
pendent programs. That is, the typical independent program 
is less concerned with teaching and educational opportunities 
as their service demands consume most of the attending staff 
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time. If, on the other hand, residents are being provided 
with educational opportunities superimposed on patient care 
obligations, the educational caliber of the program is en- 
hanced, The PMP performance of these residents may be sug- 
gestive of this type of program environment. 



Discussion and Summa ry 

The ability to determine the relative influences on 
performance indicates that the types and specific character- 
istics of residency programs are clearly significant in 
educating orthopaedists. St appears that, once an intern 
has been accepted into a program, the unique program struc- 
tures determine subsequent performance to a greater extent 
than the characteristics with which the resident entered 
the program. This is undoubtedly due, in part, to the pre- 
selection of residents which results in a relatively homo- 
geneous group of people. Having followed the same progres- 
sion from college to medical school, to surgery, and 
finally, into orthopaedics, it is not particularly surpris- 
ing that the personal characteristics of the residents are 
less influential in prediction of criterion performance 
than are program characteristics. To the extent that these 
personal characteristics are still operating to influence 
performance, the question becomes one of fitting the pros- 
pective resident and program together for the optimal 
benefit of each. 

With respect to significant program characteristics, 
the suggestive trend indicates that subsequent performance 
is more associated with selection criteria and the educa- 
tional opportunities than it is with the size or professional 
orientation of the attending staff, or with the structuring 
of the curriculum. Rather, it appears that, with respect 
to OITE performance, residents in programs with selection 
criteria which emphasize intellectual endeavors, de-emphasize 
technical aptitude, and a program which allows residents 
educational and surgical opportunities perform better on 
the OITE tests. Thus, it appears that the attitudinal set 
of the attending staff with respect to educational experi- 
ences is a better predictor of subsequent achievement than 
the size and specific structures of the program. 
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It also seems evident from this analysis that the influ 
ences on "book learning" performance arc more discernible 
than those on problem solving ability. This is probably not 
surprising, given the relatively structured nature of the 
multiple choi.ce section of the OITE as contrasted with the 
relatively less structured interpretive PMP section of the 
examination. Clearly, the more defined and concrete the 
performance criterion, the more. precise the prediction of 
the performance. However, it does not necessarily follow 
from this that one ability is more desirable than the other, 
only that there are differential influences on the two abili 
ties. Thus, in evaluating the success of an orthopaedic 
program, both of these components of orthopaedic competence 
should be considered, as well as attempts to assess the sur- 
gical competence of the residents. 



